IBM Visualization Data Explorer Programmer's Reference

Chapter 8. Importing Data

Partial Table-of-Contents

If you want to import data that is not in a format supported by Data Explorer, you have three options:

• Write an import filter to convert the data to Data Explorer or General Array importer format on disk.
• Write an import filter to convert the data to Data Explorer or General Array header format on standard output, and use the external filter option of Import to import the data. (See Import in IBM Visualization Data Explorer User's Reference.)
• Write your own import module to read the data and create a Data Explorer Object as its output (e.g., a Field Object).

Notes:

1. The following examples illustrate the conversion of data from Hierarchical Data Format (HDF) to Data Explorer file format. Understanding the examples does not require any familiarity with HDF.
2. HDF libraries are not distributed with Data Explorer and no makefiles are provided for the programs used. If you have the HDF libraries, you can use the same compilation and linking procedures as you do for other programs requiring those libraries.
3. The Import module will import HDF data. The example in 8.2 , "Writing an Import Module" is for illustration only.

8.1 Writing a Filter

The filters used to create a Data Explorer format file on disk and on standard output are essentially the same.

Assume a single data set of scalar data stored in an HDF file. All HDF files are gridded. The dimensionality and size of the grid are to be determined from queries to the data set.

The following C program requires the HDF file name as an argument. It is found in /usr/local/dx/samples/program_guide/simpleimportfilter.c .

01
02
03  #include <stdio.h>

df.h is a necessary include file for HDF library routines.

04  #include <df.h>
05
06  #define MAXRANK 3
07
08  main(argc, argv)
09    char *argv[];
10  {
11    FILE *in;
12    char filename[80];
13    int dims, counts[MAXRANK], numelements, i, j;
14    float deltas[MAXRANK*MAXRANK], origins[MAXRANK], *databuf=NULL;
15

Check that the user has supplied the name of the file to be opened.

16    if (argc < 2) {
17      fprintf(stderr,"Usage: simpleimportfilter <filename> \n");
18      return 0;
19    }
20
21    strcpy(filename, argv[1]);
22    }

The HDF library routine DFishdf checks the file for accessibility and for the correct (HDF) format. If the file is not accessible or is not an HDF file, the routine generates an error message.

23  if (DFishdf(filename) != 0) {
24      fprintf(stderr,
25              "file \"%s\" is not accessible, or is not an hdf file\n",
26              filename);
27    return 0;
28  }

Initialize the HDF library.

29  DFSDrestart();

The HDF library routine DFSDgetdims returns the dimensionality of the grid (1D, 2D, etc.) in dims. The number of positions in each dimension is returned in the Array counts.

30   DFSDgetdims(filename, &dims, counts, MAXRANK);

Determine the number of elements in the data Array.

31   numelements=1;
32   for (i=0; i<dims; i++)  {
33      numelements= numelements * counts[i];
34   }

Create a buffer for the data.

35   databuf = (float *)malloc(numelements*sizeof(float));
36   if (!databuf) {
37     fprintf(stderr,"out of memory\n");
38     return 0;
39   }

The HDF library routine DFSDgetdata reads the data from the HDF file to the data Array.

40   DFSDgetdata(filename, dims, counts, databuf);

Write the Data Explorer file format description of the data Array on standard output.

41   printf("object 1 class array type float rank 0 items %d data follows\n",
42          numelements);
43   for (i=0; i<numelements; i++)
44      printf(" %f\n ", databuf[i]);

Set the dependency of the data to "positions."

45   printf("attribute \"dep\" string \"positions\"\n ");

Now create the position origin and deltas (origin 0 and deltas 1 in each dimension).

46   for (i=0; i<dims; i++) {
47      origins[i] = 0.0;
48      for (j=0; j<dims; j++) {
49        if (i==j)
50          deltas[i*dims + j] = 1.0;
51        else
52          deltas[i*dims + j] = 0.0;
53      }
54   }

Write out the connections and positions.

55   switch (dims) {
56     case (1):
57      printf("object 2 class gridconnections counts %d\n", counts[0]);
58      printf("object 3 class gridpositions counts %d\n", counts[0]);
59      printf(" origin %f\n", origins[0]);
60      printf(" delta  %f\n", deltas[0]);
61      break;
62     case (2):
63      printf("object 2 class gridconnections counts %d %d\n",
64              counts[0], counts[1]);
65      printf("object 3 class gridpositions counts %d %d\n",
66              counts[0], counts[1]);
67      printf(" origin %f %f\n", origins[0], origins[1]);
68      printf(" delta  %f %f\n", deltas[0], deltas[1]);
69      printf(" delta  %f %f\n", deltas[2], deltas[3]);
70      break;
71     case (3):
72      printf("object 2 class gridconnections counts %d %d %d\n",
73              counts[0], counts[1], counts[2]);
74      printf("object 3 class gridpositions counts %d %d %d\n",
75              counts[0], counts[1], counts[2]);
76      printf(" origin %f %f %f\n", origins[0], origins[1], origins[2]);
77      printf(" delta  %f %f %f\n", deltas[0], deltas[1], deltas[2]);
78      printf(" delta  %f %f %f\n", deltas[3], deltas[4], deltas[5]);
79      printf(" delta  %f %f %f\n", deltas[6], deltas[7], deltas[8]);
80      break;
81     default:
82      printf(stderr,"dimensionality must be 1D, 2D, or 3D");
83      return 0;
84   }

Write out the description of the Field.

85   printf("object 4 class field\n");
86   printf("  component \"data\" value 1\n");
87   printf("  component \"connections\" value 2\n");
88   printf("  component \"positions\" value 3\n");
89
90   return 1;
91

[Data Explorer Home Page]