XML Wildcard Data Mapping and Transformation in MapForce

MapForce 2014 now supports mapping of XML wildcards that create a designed-in extension mechanism for XML Schemas, implemented as <xs:any> or <xs:anyAttribute>.

Consider this XML fragment viewed in XMLSpy:

This small portion of a very long file shows two track points recorded during a road trip by a Garmin GPS device. The XML data follows the GPX XML Schema, with Garmin extensions to add speed and course heading values. XML wildcard mapping lets you map these extensions in MapForce just like any other elements explicitly defined in the XML Schema.

Let’s assume we want to read the .gpx file and extract the highest speed recorded for a trip:

Result Caching Accelerates Application Response Time

Rapid response to user input is critical to the success of any Web application. FlowForce Server administrators can leverage result caching, a new feature introduced in version 2014, to deliver nearly instantaneous results to users running FlowForce Server jobs in a browser window as HTTP services.

Configuring a FlowForce Server job to take advantage of result caching is a simple two-step process. As an example, let’s look at the job we created recently in FlowForce Server Jobs as HTTP Services. We defined a StyleVision Server transformation as the last step in a FlowForce Server job and made the job available as an HTTP service that delivered the HTML result to a Web browser window.

Learn About XML Schema 1.1

Coinciding with support for XML Schema 1.1 validation and editing in XMLSpy 2014, we’ve released a brand new Altova Online Training course on XML Schema 1.1 technology. The course covers the fundamentals of the XML Schema language and is appropriate for beginners as well as experienced XSD developers who want to learn the ins and outs of the latest version of the standard.




This self-paced, online course is broken into nine chapters, which you may complete in order or simply focus on the areas that interest you:

1. Schema Languages
2. Schema Core Concepts
3. Built-in Datatypes
4. User-defined Simple Types
5. Complex Types
6. Namespaces
7. Identity Constraints
8. Include / Import / Override
9. New Features in XML Schema 1.1

The XML Schema 1.1 Technology course provides a balanced mix of narrated content, hands-on practice exercises, and video demos, helping you learn about XML Schema in your preferred style.
Like all Altova Online Training courses, the XML Schema 1.1 training is provided 100% free of charge. Simply access the class on our training page and complete it at your own pace.


What course would you like to see next? Leave us a comment here on the blog!

Altova Software Version 2014 is Available

It’s time for Altova’s annual major release, and this year, we’ve added support for the latest version of major W3C standards: XML Schema 1.1 and XPath, XSLT, and XQuery 3.0. Validation and processing of these standards in Altova MissionKit products is Powered by RaptorXML, bringing the benefits of Altova’s third-generation XML and XBRL engine to your favorite developer tools.
Integration for relational databases has been extended, as well, with compatibility provided for new database versions and database vendors. Support now includes:

• SQL Server® 2012
• PostgreSQL 9.0.10, 9.1.6, 9.2.1
• IBM DB2® 9.5, 9.7, 10.1
• Informix® 11.70
• MySQL® 5.5.28
• Sybase® ASE 15, 15.7
• Access™ 2010, 2013

We’ve also added numerous features based on customer requests, such as reporting multiple validation errors at once in XMLSpy, support for XML wildcards in MapForce data mapping projects, and new options for rendering section breaks in StyleVision. An overview of the new features follows, but for all the details, head over to the Altova What’s New page.
New in XMLSpy® 2014

• XML Schema 1.1 support
• Powered by RaptorXML for lightning-fast validation and processing
• Smart Fix validation with auto-correction now extended to Schema View
• XPath 3.0 support
• XSLT 3.0 (subset) support
• XQuery 3.0 (subset) support
• Support for XBRL Formula and XBRL Concept Types
• Ability to display multiple validation errors at once
• Support for new databases
• Integration with Eclipse 4.3
• Ability to generate sample XML instances with all schema choices

New in MapForce® 2014

• Support for XML wildcards (xs:any and xs:anyAttribute)
• Support for comments and processing instructions in output XML
• Integration of RaptorXML
• New function to calculate age based on a birthdate
• Support for new databases
• Integration with Eclipse 4.3

New in StyleVision® 2014

• Support for new databases
• Integration with Eclipse 4.3
• New option for rendering section breaks - ability to avoid a blank first page

New in Authentic® 2014

• Support for new databases
• Integration with Eclipse 4.3

New in UModel® 2014

• Support for project-related SPL templates
• Support for new databases
• Integration with Eclipse 4.3

Get complete details and upgrade to Version 2014 now! If you have active Support and Maintenance, this is a free software update.

Version 2014 of Altova Server Software is also available for download, with new features including support for caching results in FlowForce Server.

FlowForce Server Jobs as HTTP Services

FlowForce Server administrators can define jobs as HTTP services to empower end users to execute the job on demand, as easily as opening a Web page.

When a FlowForce Server job runs as an HTTP service results are delivered back to the Web browser. If the last step is a StyleVision Server transformation, the job can create a rich HTML-based Web page.

Even better, the same result is simultaneously saved in the enterprise workflow.

XPath Expressions for Data Reporting

In our earlier post titled Use XPath Expressions to Refine Data Selection, we described how to use XMLSpy to develop an XPath expression to select one table of data contained in a much larger data set provided by the US Department of Education.

We can reuse the work in XMLSpy to quickly create a StyleVision design for a report or an e-Form to communicate highlights from the data.

XBRL US National Conference in Las Vegas

If you will be in Las Vegas next week for the 5^th annual XBRL US National conference, be sure to check out the XBRL Data Forum for IT Pros & Developers. Altova is sponsoring this one day workshop that is part of the conference which consists of an intensive, practical day of discussion and presentation on XBRL data, databases and opportunities for IT professionals interested in expanding their technical knowledge in the growing area of XBRL.

Along with RaptorXML + XBRL, the newest addition to our product family, Altova has a number of tools for processing and validating XBRL data such as XMLSpy, MapForce and StyleVision.

Stop by booth 535 next week at Oracle OpenWorld

September here at Altova means ramping up for a busy Autumn, beautiful cool crisp days in New England and… flying out to San Francisco for Oracle OpenWorld 2013! If you are planning on attending come by booth #535 in Moscone South and see Altova from September 23 – 25^th for a demo of Altova’s tools for Oracle Users and to see our new line of cross-platform server software products: RaptorXML Server, FlowForce Server, MapForce Server and StyleVision Server. See first-hand how these new products offer high-speed automaton for projects designed using familiar Altova MissionKit developer tools.

We would love to hear from you about your latest projects and challenges, collaborate on best practices or let us show you some of the new exciting things Altova has to offer. While you are at our booth mention this blog post to receive a special giveaway. Hope to see you next week in San Francisco!

Use XPath Expressions to Refine Data Selection

In this era of big data we often need to simplify analysis and communication by creating customized views into sections of a larger file. XPath, short for XML Path Language, is designed to allow users to address parts of large XML documents. XMLSpy supports XPath with an XPath Analyzer window and in its interface for charting XML data, MapForce supports XPath functions for XML data mapping, and StyleVision supports XPath in conditional templates, extension templates, and template filtering.

Altova tools implement the same user interface wherever possible, so skills developed in one tool are easily transferrable. We’ve blogged before about XPath autocalculate in StyleVision, so let’s look at a different example.

Very often a large dataset contains a number of individual tables, such as this example of information about employment of recent college graduates from the US Department of Education, displayed here as a CSV file:

Filesystem Commands and More Wizardry with FlowForce Server Built-in Functions

In our earlier posts on FlowForce Server, we described jobs that execute MapForce Server for data transformations, StyleVision Server for report and document rendering, and RaptorXML Server for XML processing. In each scenario we also used built-in system functions to copy move, or delete files, as is often required in real-world workflows.

In this post we’ll take a look at more of the built-in functions automatically installed in the FlowForce Server system container. You can use these commands as execution steps to automate the file housekeeping so often required in enterprise production.

Constant Quest for Efficiency

In last week’s blog post titled FlowForce Server Supports RaptorXML, we created a FlowForce Server job that defined a RaptorXML execution step to validate XML files, and we called that job as a step at the end of our job that processes camera GPS data, as a final check on the output. That was a quick way to demonstrate integration of FlowForce Server and RaptorXML Server, but for real-world production we would want to perform the same task more efficiently.

If we insert the RaptorXML validation function at the top of the job, right in front of the On error definition as shown above, we can apply the same error handling steps for failures of either the data mapping or the validation step. In other words, an error in any one of a series of steps before the On error definition forces the job to take the error path.

FlowForce Server Supports RaptorXML

Starting with the latest version launched on June 27, Altova FlowForce Server now supports RaptorXML Server and RaptorXML+XBRL Server.

Altova RaptorXML is the third-generation, hyper-fast XML and XBRL processor from the makers of XMLSpy. RaptorXML is built from the ground up to be optimized for the latest standards and parallel computing environments. Now FlowForce Server jobs can include steps to validate XML, transform XML with XSLT, execute XPath and XQuery, and even perform complex XBRL operations for financial reporting including validating XBRL taxonomies, and validating XBRL instances against XBRL taxonomies with support for XBRL Dimensions and XBRL Formula, which define new, compatible functionality to extend XBRL 2.1.

For example, we can return to the FlowForce Server job described in Taming Bad Input Data with FlowForce Server and add a RaptorXML operation to validate the XML files created from raw GPS data captured by a digital camera.

XBRL Validation and Processing Tools

When we were designing RaptorXML Server, we knew from the get-go that the engine should support XML and XBRL standards. The former standard is ubiquitous and the latter will be, as well, given increasing mandates from governments worldwide as well as adoption by enterprises for analyzing financial data in a standards-based, cost-effective manner. However, we knew that at this time a sub-set of customers would be interested in the XBRL capabilities of the engine, so we created two versions: RaptorXML Server and RaptorXML+XBRL Server.



Let’s take a look at some of the powerful XBRL-oriented features in RaptorXML+XBRL Server.

Download Example Files for FlowForce Server, MapForce, and StyleVision

In response to an interested reader’s suggestion, the Altova MapForce mapping files and StyleVision stylesheet we deployed to FlowForce Server for the job described in the blog post Taming Bad Input Data with FlowForce Server are now available for download on the Altova Web site at www.altova.com/documents/AltovaBlogExampleFiles.zip

Simply unzip the archive into a new folder and you’ll have all the data mappings, the stylesheet, and other supporting data files all in one place. A ReadMe file explains the contents. You can download fully functional free trials of Altova MissionKit and FlowForce Server at http://www.altova.com/download.html and implement and test FlowForce Server yourself.

Or, execute the data mappings in MapForce and the stylesheet in StyleVision to see how easy it is to extract meaningful information from the GPS data recorded by your own digital camera or GPS device. The example files in the download were also used in Web Service as a Look-Up Table to Refine GPS Data, XPath Enhances XML Reports, and others in our series on working with XML and Global Positioning Systems.

If you’re already an Altova MissionKit user, you can download these files with examples of Web services and user functions for MapForce, and XPath calculations and chart features for StyleVision, and add them to the extensive libraries of MapForce and StyleVision samples installed with Altova MissionKit tools.

Taming Bad Input Data with FlowForce Server

Whenever you accept data from an outside source you risk encountering errors. We have blogged about this phenomenon in the past in Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery and in the series of posts on Processing the Groupon API.

Bad data in an input file can cause the data transformation step of a FlowForce Server job to fail. When a FlowForce Server Job fails, further execution steps will not be performed. FlowForce Server is designed this way to prevent an error in one job step from cascading into a series of additional invalid results. Happily, FlowForce Server also includes features to help you recover from errors and keep production flowing.

In this post we will further extend the data mapping and report rendering job described in Customizing a FlowForce Server Job to gracefully handle bad data in an input file.

RaptorXML is here!

We are thrilled to announce general availability of RaptorXML®, Altova’s third-generation XML and XBRL processing engine, for download and purchase.
Named for the dinosaur famous for its incredible speed and agility, RaptorXML provides blazing-fast validation and processing of XML, XSLT, XQuery, and XBRL data. Because it’s optimized for parallel computing on multi-core, multi-CPU machines, RaptorXML can scale quickly based on the amount of data you need to process.

Interface Options

In addition to numerous hyper-performance features and strict standards conformance, RaptorXML offers flexible implementation options for developers. These include command line operations, Java and COM APIs, a Python interface, and a built-in HTTP server. The built-in HTTP server allows RaptorXML to be integrated into SOA infrastructures and accept validation jobs, XSLT transformation jobs, etc., submitted via HTTP by other services or client applications.

Workflow Integration

RaptorXML may also be run as a component of Altova FlowForce Server, which provides a user-friendly Web interface for managing workflows based on customizable time or event triggers. This way you can automate validation or XSLT processing jobs based on a variety of triggers, or even pipeline RaptorXML jobs into multi-step processes utilizing MapForce Server and StyleVision Server that start with data aggregation, validation, and conversion and end with multi-channel report generation. The possibilities are many.

Product Options

There are two versions of RaptorXML available:

- RaptorXML Server supports validation and processing of: XML 1.0 & 1.1, XInclude 1.0, Xlink 1.0, XML Schema 1.0 & 1.1, XPath 1.0, 2.0 & 3.0, XSLT 1.0, 2.0 & 3.0 (subset), XQuery 1.0 & 3.0, and more.

- RaptorXML+XBRL Server supports all the features of RaptorXML Server, with the addition of processing and validating the XBRL family of standards: XBRL 2.1, XBRL Dimensions, XBRL Formula 1.0, XBRL Functions, and XBRL Definition Links.

A note to current customers of AltovaXML Reporting Edition: As Altova's third generation XML processing engine, RaptorXML Server replaces AltovaXML, which is being discontinued. To help existing AltovaXML Reporting Edition customers transition to the new engine and get the full value from their Support and Maintenance Package (SMP), Altova is offering all AltovaXML Reporting Edition customers with current SMP an additional license for RaptorXML Server for the remaining duration of their SMP at no cost. In fact, the value of this new RaptorXML license is significantly higher than the list price of AltovaXML. The RaptorXML Server license is for four cores and can be used in addition to the AltovaXML Reporting Edition license you currently have. To upgrade visit this page.

Free Trial

RaptorXML Server and RaptorXML+XBRL Server are available for Windows, Linux, and Mac OS, and you can download a free trial for your preferred OS.

Come see us at TechEd in New Orleans!

Altova will be exhibiting in New Orleans, LA from June 3-6 at Microsoft TechEd in booth #429. Come on down to the Morial Convention Center and speak with our experts about our new line of cross-platform server software products: FlowForce Server, MapForce Server and StyleVision Server.

If you are attending TechEd, stop by our booth to see first-hand how these new products offer high-speed automation for projects designed using familiar Altova MissionKit developer tools. While you are at our booth mention this blog post to receive a special giveaway. We can’t wait to see you next week in New Orleans!

Customizing a FlowForce Server Job

In our earlier post titled Automate Data Mapping and Transformation with FlowForce Server, we created a job called SimpleMapAndTransform to automate data mapping with MapForce Server and creation of html reports by StyleVision Server. After the FlowForce Server job ran several times, we have accumulated many output files in the same folder we use to process input files, as well as temporary intermediate files in the workFiles folder, as seen in the image below.

In this post we will enhance the job to create more orderly results and remove unneeded temporary files.

Automate Data Mapping and Transformation with FlowForce Server

Altova FlowForce Server, launched on April 29, 2013, includes numerous enhancements over earlier beta releases and one of the most exciting new features is the implementation of StyleVision Server. Now a FlowForce Server job can automate a complete data transformation workflow by executing MapForce Server for data mapping and pipelining results to StyleVision Server to render a variety of output formats.

This post describes a straightforward FlowForce Server mapping and transformation job as illustrated in these messages from the FlowForce Server Log, with the most recent step at the top of the list:

New Server Products for Data Transformation

On the heels of our RaptorXML announcement earlier this month, we are very excited to tell you about the availability of three more new products that round out the Altova server software product family.

These high-performance server products are optimized for multi-CPU servers and are highly complementary when used together for defining and managing information pipelines. Cross-platform support is immediately available for Windows and Linux, with MacOS support expected just around the corner in May. Supported languages include English, German, Spanish, and Japanese.

Let’s take a look at each of these new products in detail.

FlowForce Server

After receiving valuable customer feedback during the beta period, we have announced general availability of FlowForce Server, a powerful new tool for managing multi-step, enterprise-level data aggregation, processing, and reporting tasks.

The straight-forward FlowForce Web browser interface makes it easy to configure and monitor jobs, whether they require a single step, or multiple cascading steps starting with data aggregation and conversion, and completing with multi-channel report generation.

Announcing RaptorXML, the Hyper-fast XML and XBRL Server

 

We are very excited to be at the XBRL 26 Conference in Dublin, Ireland today to announce a brand new server product in the Altova family of XML and XBRL tools!

Altova RaptorXML is a hyper-fast XML and XBRL validation and processing server. It’s Altova’s third-generation XML and XBRL engine, built completely from scratch to help organizations efficiently validate, process, transform, and query the vast and ever-increasing amounts XML and XBRL data being generated as a result of XBRL compliance regulations and myriad other big data trends.

RaptorXML is written to be highly scalable for today’s multi-CPU and multi- core computers and servers. This, along with high performance code optimizations and an extremely low memory footprint, has helped make RaptorXML a lightning-fast XML and XBRL server that can meet the demands of today’s data processing applications.

Simply put: we architected RaptorXML to combine the performance benefits afforded by modern parallel computing environments with strict compliance to the latest versions of all relevant XML and XBRL standards.

RaptorXML includes support for the very latest versions of all relevant standards and has been submitted to rigorous regression and conformance testing. The server will be available in two versions, both of which are available for Windows, Linux, and MacOS platforms.

RaptorXML Server supports validation and processing of:

• XML 1.0 & 1.1
• XInclude 1.0
• Xlink 1.0
• XML Schema 1.0 & 1.1
• XPath 1.0, 2.0 & 3.0
• XSLT 1.0, 2.0 & 3.0 (subset)
• XQuery 1.0 & 3.0
• And more

RaptorXML+XBRL Server supports all the features of RaptorXML Server, with the addition of processing and validating the XBRL family of standards:

• XBRL 2.1
• XBRL Dimensions
• XBRL Formula 1.0
• XBRL Functions
• XBRL Definition Links

Developers creating solutions using Altova MissionKit XML development and XBRL development tools will be able to power their server applications with RaptorXML for hyper-performance, increased throughput, and efficient memory utilization, giving them the opportunity to validate and process large amounts of XML or XBRL data cost-effectively. 

Check out the complete list of supported XML and XBRL standards and more details on this groundbreaking new server product.

RaptorXML will be available to download and purchase in May.

Editing, Converting, and Generating JSON

As the use of JSON as a data transport protocol increases, I thought it would be useful to take a look at JSON support in XMLSpy. There’s been much debate about advantages of JSON vs. XML, but when you boil it down, there are simply some cases for which JSON is the best choice, and others where XML makes more sense. This article on the XML Aficionado blog unpacks this topic quite well.
While you might need to choose between JSON and XML depending on the development task at hand, you don’t have to choose between code editors – XMLSpy supports both technologies and will even convert between the two. Let’s take a look at how that works.

Web Interface Simplifies Automation of Data Transformations

FlowForce Server is a new product designed to provide automation of data transformations performed by dedicated high-speed servers. FlowForce Server can start jobs based on a variety of triggers, runs multiple jobs simultaneously, and can even run multiple instances of the same job, depending on workflow.

Monitoring all these complex activities is critical to success in a busy production environment. The FlowForce Server Web interface includes customizable views into operations, simplifying management from anywhere on the network.

Compare Folders Fast and Efficiently with DiffDog

Three user preferences in DiffDog work together to accelerate directory differencing tasks into hyperdrive, especially when you compare folders with thousands of files.

DiffDog can perform specialized comparisons. For instance, XML comparison mode commands DiffDog to determine whether two files contain equivalent XML content, even if they would be different in a text comparison because of spacing, line-ending, or attribute order variations. For the fastest directory comparisons, choose Quick Comparison mode, which only looks at file sizes and modification dates.

When you compare large directories, most often you want to quickly locate differences. Choose the Hide Equal setting to avoid listing all identical files.

Show/Hide empty directories should be set to Hide, otherwise subdirectories that contain identical files will be listed, but since their contents are equal, none of the files are shown.

After you choose settings and click the Compare button, DiffDog flies through the directory comparison.

When the comparison is finished DiffDog displays a concise listing showing only the differences, with a summary message at the bottom of the window. The toolbar in the screenshot below illustrates the settings for Quick Comparison on, Show Equal files off, and Show Empty directories off.

DiffDog saves your preferences to apply to the next folder comparison, and DiffDog even includes a quick synchronization tool to merge folders in either direction.

Find out for yourself how fast DiffDog can run your folder comparisons – download a fully-functional, free 30-day trial!

Automate Data Transformation with FlowForce Server

Altova designed FlowForce Server to provide comprehensive automation, management, and control over data transformations performed by dedicated high-speed servers. FlowForce Server Beta 3 is currently available to users of MapForce Enterprise and Professional Editions at no charge during the beta test period.

FlowForce Server can provide hot folder automation of data mappings and maintains a detailed activity log users can monitor remotely in a Web browser window. The screenshot below shows the log for FlowForce Server running the MapForce data mapping CameraLogToGPX we wrote about in the blog post titled Process Multiple Input Files in a Single Data Mapping. This mapping used wildcards to specify multiple input files for processing.

It only takes a few minutes to set up, run, and review the results of jobs like this on FlowForce Server.

Process Multiple Input Files in a Single Data Mapping

Typical data conversion tasks require processing numerous input files that arrive in batches. Altova MapForce includes features that let you handle groups of files with minimal intervention.

For instance, we recently copied a set of files from the memory card of a digital camera with GPS support. Each .LOG file is a CSV containing GPS coordinates for a single route.

We quickly designed a mapping to convert the CSV data to XML-based .gpx format and processed all three files to generate three output files in a single execution:

Generating a Schema from Multiple XML Instances

I was recently talking with one of our Support Engineers about common questions he receives from customers. One that comes up a lot is schema generation. XMLSpy will generate a valid XSD from a DTD, a relational database structure, and of course, an XML instance. But what many customers don’t know is that the XML editor will also create an XML Schema based on a group of related XML files – and this is a common requirement.

 

Let’s take a look at how it works.

 

Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery

Every time you receive data from an outside source, there is a chance it won’t arrive in the form you expect. This can require special accommodations for the rare and unlikely to make a real-world data mapping and transformation solution robust and reliable.

We processed literally dozens of .gpx files, containing hundreds of coordinates each, through the MapForce mapping we wrote about in the blog post Web Service as a Look-Up Table to Refine GPS Data. Then one day we ran a new file and encountered the error below, which caused the mapping to fail:

Reaching into the Altova MissionKit to combine features of MapForce and XMLSpy, we quickly diagnosed the issue and developed a solution we can also reuse in future mapping projects.