Q100038: Using FileGroups with FileList parameters

Before we get into the details about FileGroups and how we can utilize them with both the core modules provided with JobServer.NET and any of your own custom modules that you may create and use, we are just going to have a quick introduction to the terminology that we use when working with this type of data.

Terminology

Many of the core JobServer.NET modules will return a list of files as an output parameter. The output parameter is almost always named FileList for modules that return a single list of files. The most common example of this is the [Files] Find module, which is used to search for and select files from a specified location on the machines filesystem. The first thing to note when working with these lists of files, is that the value is always expected to contain the fully qualified pathname for the file. The pathname consists of the folder, or also called just path, in addition to the filename for the file. Consider the following pathname which has this value:

C:\Temp\Photos\IMG-1000.jpg

This is an example of a fully qualified pathname. Note that it contains the complete folder, or path, in which the file is located within. Below we see just the folder portion of the pathname. All of the JobServer modules will use the terminology of referring to this as the folder. The folder tells us where in the machines local filesystem we can locate the file that we are interested in working with.

C:\Temp\Photos

The end portion of the pathname from the above example is shown below. This is just the filename portion of the value. The filename alone does not tell us how to access the file. The filename should hopefully tell us something useful about the files contents, but for the moment we are just interested in making our terminology clear for the parameters on the various modules.

IMG-1000.jpg

The examples above all use local filesystem paths and pathnames that would be used on a machine running JobServer. Also note that when a machine that is running JobServer is part of a network domain, or has applied network domain credentials, then the path of interest could be located somewhere on the network and not part of the machine's local filesystem. In such cases, all of the terminology shown in the examples previously are still the same. The only difference is that a network path looks slightly different from a local filesystem based path. This makes it very easy to identify that any given pathname is either a local pathname or a network pathname. Look at the following example network folder (path):

\\NETWORKSRV\MrktingDept\ImageFiles

In this example, we see what would be called a UNC path. For your reference, UNC is an acronym for Universal Naming Convention. The UNC path is an accessable location on the network that a machine would be able to connect to, provided they had network credentials with the proper authorization. The full pathname for a filename using a UNC path would look like this:

\\NETWORKSRV\MrktingDept\ImageFiles\IMG-1000.jpg

Creating and Using Lists of Files

All of the modules that can work with a list of input files, are capable of doing so as long as the list of files supplied consists of any one or more files with a fully qualified pathname. Each pathname must be separated by a line terminator. The line terminator should always be a set of CR + LF characters. The majority of the core JobServer.NET modules that can work with a list of input files, will also gracefully accept an empty value with no files, as an indication that there are no files to process and will execute normally without any warnings or errors. Your custom modules should follow this same behavior when possible. However, there certainly may be cases when it makes sense that a specific module implementation must expect one or some specific number of files for processing. This is left up to the implementor of the module to determine. Thus a list of input files can potentially be zero, one, or any number of files in the list. The example below shows us what a valid list of files can look like for a modules input parameter.

C:\Temp\Photos\IMG-1000.jpg
C:\Temp\Photos\IMG-1001.jpg
C:\Temp\Photos\IMG-1002.jpg
C:\Temp\Photos\Museum\IMG-1003.jpg
C:\Temp\Photos\Museum\IMG-1004.jpg
C:\Temp\Photos\Museum\IMG-1005.jpg
C:\Temp\Photos\Museum\IMG-1006.jpg
C:\Temp\Photos\Vacation\IMG-1007.jpg
C:\Temp\Photos\Vacation\IMG-1008.jpg
C:\Temp\Photos\Vacation\IMG-1009.jpg
C:\Temp\Photos\Vacation\IMG-1010.jpg

Hopefully you take more photos on your trips than this example has, but this should be all we need to illustrate how the modules can work with this list data of valid pathnames. This list of eleven files are all in the same base root folder (C:\Temp\Photos), but eight of them are in two separate subfolders. To the modules, since it has the fully qualified pathname for each file, it can simply iterate through the list and process each file as it needs to. For most modules this is fine and it does not need to be concerned with the fact that the folder each of the files is located in, might be different than the next one. If you are passing in a list of files for a module to process, this is a completely valid and usable format to use.

FileGroups

Now that we have outlined what the basics are when working with lists of files, we can now review what FileGroups are and how they can be beneficial to certain JobServer.NET modules. FileGroups are taking the same data that we have just been looking at, but provides it in a structured format that allows certain modules within JobServer to support additional features or functionality. The default structure is a JSON based version of the list of files that provides this same data as well as some additional related data that is useful to certain modules.

Instead of the [Files] Find module actually emitting a plaintext list of files as we saw in the previous section, it actually emits this data as this FileGroups data structure. If the [Files] Find module outputs the list of files as shown in the previous section, a consuming module would not be able to tell from the data, what the base folder for this list of files is. It could have been C:\, C:\Temp, or C:\Temp\Photos. There is no way to be certain by looking at that data. You may be asking the question, why would this be important? Well, if a module that consumes these files wants to be aware of the folder hierarchy, then this becomes a key piece of information. An example of this is using the [Files] Copy/Move module. One of the options that the module supports is the Replicate subfolders option. By knowing what the base folder is, we now have a point of reference to know what the relative pathname is based on the difference between the base folder and the fully qualified pathnames. To see how this works, lets look at what the same data would look like if we used the FileGroups data format instead of the raw list of pathnames from the previous section.

{
   "DataType":"FileGroups",
   "DataVersion":"1.0",
   "BasePath":"C:\Temp",
   "Parser":"Unstructured",
   "Groups":[
      {
         "Path":"C:\Temp\Photos",
         "Files":[
            "C:\Temp\Photos\IMG-1000.jpg",
            "C:\Temp\Photos\IMG-1001.jpg",
            "C:\Temp\Photos\IMG-1002.jpg"
         ]
      },
      {
         "Path":"C:\Temp\Photos\Car",
         "Files":[]
      },
      {
         "Path":"C:\Temp\Photos\Museum",
         "Files":[
            "C:\Temp\Photos\Museum\IMG-1003.jpg",
            "C:\Temp\Photos\Museum\IMG-1004.jpg",
            "C:\Temp\Photos\Museum\IMG-1005.jpg",
            "C:\Temp\Photos\Museum\IMG-1006.jpg"
         ]
      },
      {
         "Path":"C:\Temp\Photos\Vacation",
         "Files":[
            "C:\Temp\Photos\Vacation\IMG-1007.jpg",
            "C:\Temp\Photos\Vacation\IMG-1008.jpg",
            "C:\Temp\Photos\Vacation\IMG-1009.jpg",
            "C:\Temp\Photos\Vacation\IMG-1010.jpg"
         ]
      }
   ]
}

With the list of pathnames now made available to our module as a FileGroup, we have the potential to do a lot more with the files if we need the hierarchy of the folders to provide any meaning to the consuming module. The primary example of this is when you are looking to copy or move the files from one location to another and preserve the hierarchy of folders from the source to the target. So what is the meaning of this additional information in the FileGroup data? First, you should note that the BasePath property of the FileGroup allows the consumer of this data to be able to determine relative paths when needed from the originating source of the list of files. The pathnames always give us the fully qualified path to work with the source file. When the consuming module needs to be able to determine a relative path relationship from the source files, then it can just determine this difference from the basepath. Note that when the relative relationship of the file hierarchy needs to be preserved, this also properly supports the inclusion of empty folders as we see in this example with the folder C:\Temp\Photos\Car.

Consuming FileGroups in Custom Modules

Using the FileGroup data in the core JobServer.NET modules is completely transparent when you use the FileGroup class. You don't need to be concerned if your input list of files is a plaintext list of raw pathnames or if it is the FileGroup data structure. In a custom module, you can just instantiate the FileGroups class and pass in the list, and it will properly parse or deserialize the data for you. An example C# implementation might look something like this example:

FileGroups groupFiles = new FileGroups(inputParameter);
if (groupFiles.FileCount == 0) {
    Console.WriteLine("No files to process.");
} else {
    // Process files by iterating through the collection of files in the groups
    Console.WriteLine($"There are {groupFiles.GroupCount} folders with a total of {groupFiles.FileCount} file(s) to process.");
    foreach (FileGroup group in groupFiles.Groups) {
        Console.WriteLine($"Group Path: {group.Path}  FileCount: {group.FileCount}");
        foreach (string pathname in group.Files) {
            Console.WriteLine($"   File: {pathname}");  // Do your proccessing on this file a
        }
    }
}

Emitting FileGroups in Custom Modules

When you have a process where you want to emit a list of files. You can always just emit a list of fully qualified pathnames in a List<string> parameter and it will be recognized by all modules that implement FileGroups. If your process generates files where the folder hierarchy of the files might need to be used by a consuming module, then you can also easily use FileGroups programatically. An example C# implementation that outputs a list to a module's output parameter might look like the following example. This outputs the JSON data example shown previously.

FileGroups groupFiles = new FileGroups();
groupFiles.BasePath = "C:\\Temp";
groupFiles.AddFile("C:\\Temp\\Photos\\IMG-1000.jpg", "C:\\Temp\\Photos");
groupFiles.AddFile("C:\\Temp\\Photos\\IMG-1001.jpg", "C:\\Temp\\Photos");
groupFiles.AddFile("C:\\Temp\\Photos\\IMG-1002.jpg", "C:\\Temp\\Photos");
groupFiles.AddFolder("C:\\Temp\\Photos\\Car");
groupFiles.AddFile("C:\\Temp\\Photos\\Museum\\IMG-1003.jpg", "C:\\Temp\\Photos\\Museum");
groupFiles.AddFile("C:\\Temp\\Photos\\Museum\\IMG-1004.jpg", "C:\\Temp\\Photos\\Museum");
groupFiles.AddFile("C:\\Temp\\Photos\\Museum\\IMG-1005.jpg", "C:\\Temp\\Photos\\Museum");
groupFiles.AddFile("C:\\Temp\\Photos\\Museum\\IMG-1006.jpg", "C:\\Temp\\Photos\\Museum");
groupFiles.AddFile("C:\\Temp\\Photos\\Vacation\\IMG-1007.jpg", "C:\\Temp\\Photos\\Vacation");
groupFiles.AddFile("C:\\Temp\\Photos\\Vacation\\IMG-1008.jpg", "C:\\Temp\\Photos\\Vacation");
groupFiles.AddFile("C:\\Temp\\Photos\\Vacation\\IMG-1009.jpg", "C:\\Temp\\Photos\\Vacation");
groupFiles.AddFile("C:\\Temp\\Photos\\Vacation\\IMG-1010.jpg", "C:\\Temp\\Photos\\Vacation");
List<string> outputParameter = groupFiles.ToJsonList();

Legacy FileGroups

Older versions of JobServer.NET supported a different structured data format for lists of files in a format known as chunked data before the current JSON based structure was adopted. While it will currently still recognize the older chunked data format, this is considered a deprecated feature and should not be used for any new module implementations. In the chunked format the first line has the identifier tag [FileGroup]. This defines what this data is and how it is going to be structured. Then next thing you should note that each line is prefixed with an identifier. Either the [D] or the [F] tags. The [D] tag tells us that this line is going to be a folder, whereas the [F] tag tells us this line is a fully qualified pathname. The sequence of the lines also has meaning, in that the files that are contained within a specific folder will always be listed after the entry for that folder. In other words, all the files located within each folder, will be found immdiately after the corresponding [D] tag for the folder. This means that empty folders can also be represented. We see that in this example with the folder C:\Temp\Photos\Car. With the FileGroup, we know that the source contains this empty folder. Additionally, we know that the first folder is the base folder that the module was working from. Therefore, if we received this data from the [Files] Find module, we know that the C:\Temp\Photos folder is the base, or FileSource, that the module is working from.

[FileGroup]
[D]C:\Temp\Photos
[F]C:\Temp\Photos\IMG-1000.jpg
[F]C:\Temp\Photos\IMG-1001.jpg
[F]C:\Temp\Photos\IMG-1002.jpg
[D]C:\Temp\Photos\Car
[D]C:\Temp\Photos\Museum
[F]C:\Temp\Photos\Museum\IMG-1003.jpg
[F]C:\Temp\Photos\Museum\IMG-1004.jpg
[F]C:\Temp\Photos\Museum\IMG-1005.jpg
[F]C:\Temp\Photos\Museum\IMG-1006.jpg
[D]C:\Temp\Photos\Vacation
[F]C:\Temp\Photos\Vacation\IMG-1007.jpg
[F]C:\Temp\Photos\Vacation\IMG-1008.jpg
[F]C:\Temp\Photos\Vacation\IMG-1009.jpg
[F]C:\Temp\Photos\Vacation\IMG-1010.jpg