Command: execute/execute-live

Purpose:

The execute command is used to send command instructions to the FlexRAID host service for Snapshot RAID.

The execute-live command is similar to the execute command except that it applies only to Real-Time RAID configurations and requires an instance ID parameter value. Although FlexRAID can support multiple storage pools, it currently restricts users to a single storage pool instance per system.

Tasks:

The core FlexRAID core engine is task based.

Format:

execute
task=
data=
parity=
engine=
... //more properties as required or needed
. //end of message

 

execute-live

live=true
task=
data=
parity=
... //more properties as required or needed
. //end of message

Examples:

execute
task=create
data={}
parity={}
engine=T1+
processes=3
.

Response:

Returns a CommandResponse object.

If the command execution was successful (CommandResponse.success == true), the CommandStatus will be set and CommandStatus.referenceCode can be used to query for the status of the running task. Additionally, CommandResponse.commandMessages will contain information about the successful operation.
Otherwise, the CommandStatus will be null. Either or both of the CommandResponse.commandMessages and/or CommandResponse.serverMessages will have details on the reason for the failure.

The FlexRAID tasks

Snapshot tasks:
- Create: initializes the RAID
- Update: synchronizes the RAID
- Fast-Expansion: synchronizes the RAID by only processing new drives with data (only the parity drives and the new drives will be active)
- Verify: validates the RAID configuration (validates by checking every single bit – very slow and most reliable)
- Force-Sync-Verify: validates the RAID configuration and synchronizes any bit mismatch to the parity data
- Validate: validates the RAID configuration (improves on Verify by validating through algorithms instead of bit for bit checks)
- Quick-Validate: validates the RAID configuration but does not detect datarots
- Restore: restores the data on a failed drive
- Migrate: migrates the FlexRAID metadata database format if it changes
- Multi-Step-Create: see http://wiki.flexraid.com/2011/03/24/multi-step-parity/
- Multi-Step-Restore: see http://wiki.flexraid.com/2011/03/24/multi-step-parity/

Real-Time tasks:
- Create: initializes the RAID
- Fast-Expansion: synchronizes the RAID by only processing new drives with data (only the parity drives and the new drives will be active)
- Fast-Contraction: removes a DRU from the parity data
- Verify: validates the RAID configuration (validates by checking every single bit – very slow and most reliable)
- Force-Sync-Verify: validates the RAID configuration and synchronizes any bit mismatch to the parity data
- Restore: restores the data on a failed drive
- span-update: adds/removes a spanned drive from a DRU or PPU
- Reconcile: Reconciles the filesystem databases of a Real-Time RAID configuration with the underlying filesystem

Other tasks (triggered through the execute command):
- copy-live-dbs: copies/restores (backup/restore) the filesystem database files for a Real-Time RAID config
- Copy-Data: copies perfectly a drive or folder to another drive or folder preserving attributes, extended attributes, dates, security information, etc.

Advanced RAID reconfiguration

Once a DRU has been processed, it is locked such that removing it from your configuration file has no effect.
If you need a locked DRU removed or renamed, you will need to direct FlexRAID to do so as explained below.

Remove:
data=DRU1{A:\Path@remove;/MyOtherPaths}

Rename:
data=DRU1{A:\old path@rename@B:\new path;/MyOtherPaths}

Though you don’t have to, you should remove the @remove and/or @rename@ notations after rsynch completes so that those directives are not uselessly processed again.
If the specified directories were already removed/renamed, the directives will simply be ignored.

Snapshot RAID Properties

 Name   Label   Advanced   Changeable   Description 
processes Processes false true The number of process threads to run the task under.
The optimal setting for this property will depend on your system.
Setting it too low or too high will hinder the processing performance.
On most systems, a value between 3 and 5 will be optimal.
Do your own testing to determine what works best for you.
This property can be altered while the task is running by the “max” command.DEFAULT: 3
bufferSize I/O Buffer Size false true I/O buffer size.
The ideal buffer size depends on your own disk configuration and cluster size.
Feel free to experiment with values greater or lesser than the default value to see what value yields the greatest performance.
Choosing the proper buffer size is crucial to optimal performance.DEFAULT: 1MB
leaveOff Parity Space Reserve false true The amount of free space to leave off each parity target.
You should leave some amount of free space to allow the parity metadata info to grow.
You can also use the property to reserve space for other use.
Eg.,
parity=PPU1{/Path1;Path2}|PPU2{/Path1}
leaveOff=50MB;0;20GB
50MB will be left off PPU1{Path1}, 0 byte on PPU1{Path2}, and 20GB on PPU2{Path1} and all other remaining paths (the last value serves as default to all remaining paths).
Warning: you should always leave some space (1 or 2MB) on each parity volume/partition due to how the various filesystems handle written data.
It is possible to get away with leaving 0 byte or just a few KB, but that widely varies per filesystem.
Users RAID’ing large amount of data (Terabytes+) should be safe and leave at least 10MB on each volume hosting the parity data.
The leaveOff property is optional and the default behavior is to leave off 10MB on each volume/partition hosting the parity data.DEFAULT: 10MB
splitSize Split Size false true The parity file split size.
The parity data should be split into multiple files so that it is easily managed.
The splitSize should be less than the maximum file size supported by your filesystem.DEFAULT: 1GB
metadata Metadata Path false true The folder path or the full path to the metadata file.
The metadata file can be found in the first parity data path unless it was specified to be placed elsewhere.
E.g., metadata=/Path/flxr.meta
Or, metadata=/Path
abortRestoreOnError Abort Restore On Error false true Whether to abort the restore process if an error is encountered.

DEFAULT: true

strictOnFreeSpace Strict On Free Space false true Whether to abort the process if the OS does not report enough space for parity.
Setting this to false is useful when using mount points that causes the OS to not properly report the true free space of the underlying storage.DEFAULT: true
disableParityValidation Disable Parity Validation true true Whether to disable datarot detection on parity data.
Parity validation can only be disabled during RAID creation time and cannot be changed afterward without re-creating the RAID.
Disabling parity validation will speed up update operations.
Disabling parity validation will not disable data validation and datarot detection on your DRUs.DEFAULT: false
disableRestoreValidation Disable Restore Validation true true Whether to disable datarot detection on restored data during data recovery.
This property can be changed any time before the operation is started.
Disabling restore validation will speed up the restore operation.DEFAULT: false
fullPreValidateOnRestore Full PreValidate On Restore true true Whether to pre-validation the remaining surving DRU before doing the restore.
This is useful for RAID configurations with multiple PPUs.
FlexRAID will swap out inconsistent DRUs (as determined by the pre-validation) with the remaining PPUs for the recovery.
checksum Checksum true true The checksum implementation to use for datarot detection and general checksum functions.
If no value is provided, FlexRAID will pick the best value (recommended).Possible values are:
adler32 (32 bit)
crc32 (32 bit)
digest Digest true true The digest implementation to use for datarot detection and general digest functions.
If no value is provided, FlexRAID will pick the best value (recommended).Possible values are (but not limited to):
GOST3411 (256 bit)
MD2 (128 bit)
MD4 (128 bit)
MD5 (128 bit)
RipeMD128 (128 bit – basic RipeMD)
RipeMD160 (160 bit – enhanced version of RipeMD)
RipeMD256 (160 bit – expanded version of RipeMD128)
RipeMD320 (160 bit – expanded version of RipeMD160)
SHA1 (160 bit)
SHA-224 (256 bit – FIPS 180-2)
SHA-256 (256 bit – FIPS 180-2)
SHA-384 (384 bit – FIPS 180-2)
SHA-512 (512 bit – FIPS 180-2)
Tiger (192 bit)
Whirlpool (512 bit)
retryReadErrors Retry Read Errors true true By default read errors are not retried and will cause the process to abort.
Setting the property to true, will cause FlexRAID to retry ignore read errors.DEFAULT: false
retryCount Retry Count true true The number of times to retry a given read error before giving up.

DEFAULT: 10

retryIntervalMilliSeconds Retry Interval (ms) true true The amount of time to wait between read error retries (in milliseconds).

DEFAULT: 100

ignoreReadErrors Ignore Read Errors true true Whether to ignore read errors.
If read errors are set to be retried, read errors will be retried before this property is evaluated.DEFAULT: false
enableUndoOperation Enable Undo Operation true true Whether to enable the option to rollback failed rsynch operation.
You must define a URU (Undo Recovery Unit) when setting the property to true.DEFAULT: false
undoLeaveOff Undo Space Reserve true true Same as the leaveOff property except that it applies to the URU.

DEFAULT: 10MB

newFileDelayMilliSeconds New File Delay (ms) true true A delay since the last modified date specifying when a new file is ready to be added to the RAID (in milliseconds).

DEFAULT: 10000 (10 seconds)

validateSet Validate Set true true The file or set of files to validate.
When not specified, all files will be validated.
Ex1: validateSet=/path1;/path2;…;/pathX (only the specified files will be included)
Ex2: validateSet=/path1* (all files starting with /path1 will be included)
Ex3: validateSet=*something/MyEndingPath (all files ending with something/MyEndingPath will be included)
Ex4: validateSet=[RegExp]MyRegularExpressionPattern (this will use regular expression for matching)Of course, you can mix and match the patterns:
Ex5: validateSet=/Path1;*.mp3;[RegExp]MyRegularExpressionPattern
verifyStart Verify Start true true Start the verify operation starting at the specified byte.
Ex: verifyStart=2048000000
verifyNumOfBytes Verify Number Of Bytes true true The number of byte(s) to verify.
If not specified when verifyStart is specified, all bytes starting from the specified start point will be verified.
Ex: verifyNumOfBytes=1000
exclusions Exclusions true true The file or set of files to exclude.
Ex1: exclusions=/path1;/path2;…;/pathX (only the specified files will be excluded)
Ex2: exclusions=/path1* (all files starting with /path1 will be excluded)
Ex3: exclusions=*something/MyEndingPath (all files ending with something/MyEndingPath will be excluded)
Ex4: exclusions=[RegExp]MyRegularExpressionPattern (this will use regular expression for matching)Of course, you can mix and match the patterns:
Ex5: exclusions=/Path1;*.mp3;[RegExp]MyRegularExpressionPattern
deletethreshold Delete Threshold true true The number of files detected as deleted past which FlexRAID should abort all update operations.
This option is useful to protect against accidental operations or virus actions.
If setting this value, set it to a value just above the number of files you see yourself deleting between RAID updates.
diffCheck Differential Check true true Whether to apply differential change detection.
FlexRAID uses a set of fuzzy logics to detect change for the purpose of RAID updates and applies them in layers.
Differential change detection is a layer FlexRAID omits by default based on statistics and due to its overhead.
On the other hand, differential change detection produces more precise change detection and will yield a much smaller RAID update size.
Based on statistics, it is more wasteful if applied to all files, but it can improve RAID updates if applied to the right set of files.
Files such as incremental backups and logs (files that grow over time)make great candidates for differential change detection.DEFAULT: true
diffCheckThreshold Differential Check Threshold true true The file size under which differential change detection should not be applied.
This setting is a first level filter to exclude files under a certain size.
Differential change detection carries a certain overhead and works best when not applied to all files and in particular small files.DEFAULT: 50MB
diffCheckMatch Differential Check Match true true The pattern of files to apply differential change detection to.
Differential change detection carries a certain overhead and works best when not applied to all files.
If applied to all files (value set to “*”), try to limit based on the file size using the threshold property.Ex1: DiffCheckMatch=* (apply to all files)
Ex2: DiffCheckMatch=/path1;/path2;…;/pathX (only the specified files will be evaluated)
Ex3: DiffCheckMatch=/path1* (all files starting with /path1 will be evaluated)
Ex4: DiffCheckMatch=*something/MyEndingPath (all files ending with something/MyEndingPath will be evaluated)
Ex5: DiffCheckMatch=[RegExp]MyRegularExpressionPattern (this will use regular expression for matching)

Of course, you can mix and match the patterns:
Ex5: DiffCheckMatch=/Path1;*.mp3;[RegExp]MyRegularExpressionPattern

DEFAULT: *.log;*.bkf;*.iv2i;*.backupdb

Real-Time RAID/Storage Pool Properties

 Name   Label   Advanced   Changeable   Description 
unique Unique Volumes false true The root directories to merge. Enter one per line.
This is needed so that if you have your drives mounted in folders like (C:Ddrive; C:Edrive; C:Fdrive), FlexRAID could understand that such paths point to unique disk partitions.
restrict Restricted Volumes false true When a volume is restricted, only data meant for that volume is allowed to be written there.
If another volume runs out of space, the space from a restricted volume cannot be used.
For instance, you might want to add your C: drive to the pool, but you might also want to restrict it so that its free space is not used when another drive runs out of space.
thread Threads false true The number of concurrent processes the filesystem should support before queueing.

DEFAULT: 5

bufferSize I/O Buffer Size false true The I/O buffer size. This parameter has no significance when writing to and reading from the storage pool.

DEFAULT: 1MB

reserve Reserve true true The reserve space should ideally be between 5-10GB (more if the average file you write is larger, less if otherwise, but not less than 2GB).
FlexRAID will fill all your drives until they each reach the “reserve” space amount before it does a more intricate disk space management.
It is not necessary to ensure that your disks have this amount of space available.
This “reserve” will be applied going forward.DEFAULT: 10GB
paritySize Parity Size true false By default, FlexRAID will use all the available space on the PPU drive for parity.
During testing or under special circumstances, it can be useful to limit the parity data to a specific size.
For instance, if your largest DRU is 2TB and your PPU is 3TB, you could restrict the parity file to 2TB so that you can use the other 1TB for other purposes.
removable Removable true true Whether to create the virtual drive as a fixed drive or as a removable drive.

DEFAULT: true

allow_root Grant access to Root true true Linux users only:
Whether to grant the root user access to the storage pool.DEFAULT: false
allow_other Grant access to others true true Linux users only:
Whether to grant all other users (other than the one mounting the storage pool) access to the storage pool.DEFAULT: false
syncFolderLastMod Directory changed date fix true true Windows and Auto modes only:
Set to true to apply a workaround when on Windows and using an Auto merge mode if you have an application that relies on directories last modified dates for its functions.
Changes require the View configuration to be republished…DEFAULT: false
filesystem FileSystem true true The name of the filesystem to show the storage pool as.

DEFAULT: FlexRAIDFS

Real-Time RAID only Properties

 Name   Label   Advanced   Changeable   Description 
paritySize Parity Size true false By default, FlexRAID will use all the available space on the PPU drive for parity.
During testing or under special circumstances, it can be useful to limit the parity data to a specific size.
For instance, if your largest DRU is 2TB and your PPU is 3TB, you could restrict the parity file to 2TB so that you can use the other 1TB for other purposes.
verifyStart Verify Start true true Start the verify operation starting at the specified byte.
Ex: verifyStart=2048000000
verifyNumOfBytes Verify Number Of Bytes true true The number of byte(s) to verify.
If not specified when verifyStart is specified, all bytes starting from the specified start point will be verified.
Ex: verifyNumOfBytes=1000

Storage Pool only Properties

 Name   Label   Advanced   Changeable   Description 
recyclebinmode Enable recycle bin mode true true Whether to move deleted files to a recycle bin and delete them only after a successful Snapshot RAID update.

DEFAULT: false

Be Sociable, Share!

No comments yet.

Leave a Reply

3 × three =