2. Home
  3. /
  4. Guide Admin
  5. /
  6. Gestion des données
  7. /
  8. Data Manager

Data Manager

The Data Manager application will only contain:

  • datamappings.config
  • datamanager.config
  • privateKey.xml
  • bin:
    • Release:
      • Files:
        • Log:
          • Verbose: In this folder you can find Data Manager log files
    • CirrusShield.DataManager.exe
    • CirrusShield.DataManager.exe.config
    • CirrusShield.Security.dll

To run the data manager, you need to follow the below steps:

  1. Prepare your .csv source data files. The files should be encoded in UTF8 and the data types in the file should respect the format described in section 3 below.
  2. Configure the datamanager.config and datamappings.config files (see section 4 of this document).
  3. Run the executable file: CirrusShield.DataManager.exe
  4. Check the output error and success files.
1. Boolean

-True values (case insensitive) = 1

– False values (case insensitive) = 0		7. Currency
2. Date Formats

– The format of Date is: YYYY-MM-DD

– The format of Date/Time: YYYY-MM-DD HH:MM:SS+TZ

– Where TZ is the timezone. Example: 2017-01-15 10:00:00+01		8. Text
3. Email9. Text Area
4. Lookup Relationship10. Text Area (Rich): Html
5. Number :

– Format: nnnnnnnnnnnnnnnn.nn		11. URL
6. Percent :

– Format: nnn.nn		 

Lookup Relationship

When importing data using the Data Manager, you can fill the “Lookup Relationship” field values, in the csv file that you want to upload into the Cirrus Shield instance, with the values of the “External Id” field of the parent object, instead of filling them with the “Cirrus Shield Id” values.

The “External Id” field is a field that is used as the unique record identifier from an external system when importing data into the Cirrus Shield instance.

To define a field as an “External Id” field, you will need to check the “External Id” property found when creating/updating a field in the Cirrus Shield platform.

https://help.cirrus-shield.com/wp-content/uploads/2017/05/Datamanager.png

In the current version of Cirrus Shield, only one “External Id” field can be defined by object.

If an “External Id” field was defined on an object, the “Lookup Relationship” fields to that object must be filled with the object “External Id” field values and not the “Cirrus Shield Id” values.

Example:

Let’s define on the “User” Cirrus Shield standard object a custom field “User External Id” of datatype “Text”, and check the “External Id” property. Let’s also create a custom object “Account”.
Then we will add 2 Cirrus Shield users having the following “Id” (Cirrus Shield Id) and “User External Id” fields values:

https://help.cirrus-shield.com/wp-content/uploads/2017/05/DataManager2.png

If we want to upload “Account” records into the Cirrus Shield instance, we will need to fill the “Account” “Lookup Relationship” fields (to the “User” object) by the “User External Id” field values and not the “Id” (Cirrus Shield Id) field values.

The csv file containing the “Account” records to upload will be as follows:

https://help.cirrus-shield.com/wp-content/uploads/2017/05/DataManager3.png

The “Account 1” record will have as owner the user: “Noemi Leroy” and the “Account 2” will have as owner the user: “Alice Robert”.

Note that if we have typed for example “123457” instead of “123456” in the “Owner Id” column of the csv file, the record will not be inserted if no user has the “User External Id” value “123457”.

This section explains the configuration files of the Data Manager. Those files are used by the Data Manager at runtime to:

  • Define the target Cirrus Shield instance in which the data is to be loaded using the specified user for the connection.
  • Define the path and files to be used as source data to be imported into the corresponding Cirrus Shield instance.
  • Define the mappings of the source fields (in the .csv files) to the target fields (in the Cirrus Shield instance)

1. Path to DataManagerConfig and DataMappingsConfig

In CirrusShield.DataManager.exe.config and CirrusShield.DataManager.vshost.exe.config files you have to change the following:

<appSettings>

<add key=”DataManagerConfig” value=”C:DataManagerdatamanager.config”/>

<add key=”DataMappingsConfig” value=”C:DataManagerdatamappings.config”/>

</appSettings>

You specify the path to datamanager.config and datamappings.config in the value of these two keys in both files listed above.

  • datamanager.config: file containing information about the Data Manager.
  • datamappings.config: Definition file of the correspondences of the fields.

2. Configuration file : datamanager.config

In this file, you have to specify:

  • The username
  • The password of the user (encrypted value).
  • The URL of the Soap Api of your application for which protocol you are using “http” or “https” (Obviously the https is strongly recommended).
  • The CSV delimiter (, or ;).
  • The bulk size (the number of records to be inserted/updated at the same time).
  • The AutoExit (true of false): indique au data manager s’il doit donner la main à la ligne de commande en terminant le chargement ou s’il doit attendre une action manuelle de l’utilisateur.

Here is an example of how to set the datamanager.config file:

<?xml version="1.0" encoding="utf-8" ?>

<configuration> 
<<span style="color: 
#993300;">appSettings> 
 
<add key="UserName" value="nom.utilisateur@domaine.fr"/>

<add key="Password" value="VALEUR_ENCRYPTEE_DU_MOT_DE_PASSE"/>

<add key="HttpCSInstance" value="http://localhost:1739/CirrusShieldWS.asmx"/>

<add key="HttpsCSInstance" value="https://ws.cirrus-shield.net/CirrusShieldWS.asmx"/>

<add key="CSPrivateKey" value="C:DataManagerprivateKey.xml"/>

<add key ="Proxy" value=""/>

<add key ="ProxyUser" value=""/>

<add key ="ProxyPwd " value=""/>

<add key ="CSVDelimiter" value=","/>

<add key="Bulk" value="true"/>

<add key="BulkSize" value="10"/>

<add key="AutoExit" value="true"/>

<add key="WebServiceProtocol" value="https"/>
 </appSettings> 
</configuration>

Note: For encryption of password you will need the encryption tool provided within the Cirrus Shield platform: EncryptDataManagerCirrusShieldUserPassword
You can find this tool in “DropboxCirrus ShieldR&DToolsEncryptDataManagerCirrusShieldUserPasswordEncryptionGUI.exe”.

3. Configuration file : datamappings.config

In this file, you can define the target object into which you plan to import your data, as well as specify the mappings between your application’s fields and the csv files fields.

Here is a sample mapping:

<map> 
 
<FilePath>C:datasourcePolicies.csv</FilePath>

<ObjectName>Policy</ObjectName>

<Action>Insert</Action>

<MatchingField>Policy_Number</MatchingField>

<Fields> 
 
<Field Column="POLICY_NUMBER" APIColumn="Policy_Number" ></Field>

<Field Column="PRODUCT" APIColumn="Product" ></Field>

<Field Column="HOLDER_NUMBER" APIColumn="Holder_Number" ></Field>

<Field Column="HOLDER" APIColumn="Holder" ></Field>

<Field Column="INSURED_NUMBER" APIColumn="Insured_Number" ></Field>

<Field Column="INSURED" APIColumn="Insured" ></Field>

<Field Column="PAYER_NUMBER" APIColumn="Payer_Number" ></Field>

<Field Column="PAYER" APIColumn="Payer" ></Field>

<Field Column="TOTAL_PREMIUM" APIColumn="Total_Premium" ></Field>

<Field Column="NUMBER_OF_PAYMENTS" APIColumn="Number_of_Payments" ></Field>

<Field Column="COMMISSION" APIColumn="Commission" ></Field>

<Field Column="UNPAID_COMMISSION" APIColumn="Unpaid_Commission" ></Field>

<Field Column="ACCOUNT_VALUE_LIFE" APIColumn="Account_Value_Life" ></Field>

<Field Column="INSURANCE_TERM" APIColumn="Insurance_Term" ></Field>

<Field Column="PAYMENT_TERM" APIColumn="Payment_Term" ></Field>

<Field Column="POLICY_VALUE" APIColumn="Policy_Value" ></Field> 
 
</Fields>

<ResultDirectory>C:dataResults</ResultDirectory>
 
 </map>

The above example is used to show the following:

1. The path of the csv file used as a source data to import.

2. Choice of the data manager action: insert, upsert, update or delete.

3. Specification of the matching field that is used as a primary key for the data.

4. Definition of the mappings between the fields in the csv file and the application.

– “Column” corresponds to the name of the field in the csv file.

– “API Column” is the fields name in your application.

5. Specification of the path to the Results folder where you would like the result information to be stored. This folder will contain a log of the results of the actions of the Data Manger. A csv file will be saved (Name: datetime-name_of_original_csv.csv) in it with the same fields in the original csv file, complemented by two additional fields: the first for the success of action of each record and the second is for the error message.

4. Actions

With Data Manager you can insert, upsert, update or delete records:

  • To insert, you have to specify from the system fields only the name of record and the Owner Id which is the user’s internal id that owns this record or the user’s external Id if it exists.
    • By default, the fields CreatedBy and ModifiedBy will be assigned to the user importing the data specified in datamanager.config
  • To update, if you don’t want to update all the fields of this record you can keep only those you want to change and the matching field.
    • You have to be careful to the matching field you choose, it should be unique if you want to update only a specific record.
    • Important: You have to keep in datamappings.config only the updated fields, if you kept all fields, the system will take the not existing fields in the csv file as null and will update them as null.
  • To upsert, you have to be careful to the matching field, if it’s not unique, data manager will update all the fields that have the same matching field value.
    This action will allow users to Update existing records and insert non existing records.
  • To delete, you have to keep only one column in your csv file. Choosing this column depends if you want to delete a specific record or a group of records.
    If you want a specific record, the only column in your csv should be a field that you are sure it’s unique.
    If you want to delete all records that have for ex. the Broker_Number = 1, the column that you have to leave in the csv file is the Broker_Number with the value 1

To run the data manager and after updating all the files as described in this guide, you have to double-click on CirrusShield.DataManager.exe.

When using the data manager, you may face some errors and problems, below you will see some of them:

General Errors in the Command Line

  1. « There is an error during the processing of the file»
    This error is by Chunk or bulk size, for ex. if you have the bulk size = 2000 and there is an error in 500 records out of 2000 you will see the error message only once for these 2000 records.
    For the second bulk records (2000 records for ex.) you will see a second error message in case of an error in one record out of 2000 and in case of an error in 2000 records out of 2000.
  2. You may have a fatal error if you have by mistake an “enter” in a field, this will cause a system error.
  3. Other error may be from the connection between the web service that the Data Manager us and the database especially if the database is installed in separate server.
  4. If the csv file of a map doesn’t exist, the Data Manager logs this info in “FilesLogverbose” and continues to the next csv file in mappings.

Errors in Results File

This error will show in case of invalid value for a field, you can see details of error in the csv generated in the results folder such as:

  • Error_MSG_Field_Required : Non-existing value for a required field
  • Error_MSG_Value_entered_not_valid: Non-valid format of a value for specific type of field
    • Examples :
      • “Date_of_Birth:Error_MSG_Value_entered_not_valid”: This message will appear if the type is for ex. date but the format wasn’t as mentioned in section 4.
      • “Email: Error_MSG_Value_entered_not_valid”: This message will appear if the type is for ex. Email but the value isn’t an appropriate Email.
  • “This Record does not exist”: This message will appear if you are updating a non-existing record.
  • “This record already exists”: This message will appear if you are inserting an existing record.
  • Some of the Picklist Fields contain invalid values:
    • This error occurred when having a picklist value in the csv file not existing in Cirrus Shield.
  • Some of the Lookup Fields contain invalid values:
    • If the record you are inserting has a field that points to another object (a “Lookup Relationship” field) and the record pointing to it does not exist, the record will not be inserted.
Was this article helpful to you? No Yes

How can we help?