User:Vidhyan: Difference between revisions

Browse history interactively

← Previous edit Next edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 22:29, 18 August 2008

FIO enhancements

Intern : Vidhya Nagarajan

Mentor : Luis Arruda

Manager : Sudheer Apte

Goal

The goal of my project is to enable modularity in FIO and Unit Test its functionality.

What is FIO?

FIO is a bunch of functions used to access WAFL files. It presents a POSIX-like API to create, open, write, etc.

These are some clients of FIO:

local configuration files
dump/restore
qtree quota checks
Coral files etc.

Scope

FIO has multiple clients. NDMP is one such client which uses FIO for dump and restore. For convergence, We are starting to unify the FIO interface for both 7-mode and 10-mode volumes. Meanwhile, we need to continue to support NDMP and all other clients of FIO. We basically need test cases to analyze how each one of these clients interact with FIO. Focusing on one client(NDMP) for now will put things in perspective.

Currently for the Boilermaker(BR) release, the NDMP dump/restore is not working the way it worked in 7G.This is because the spinnp fileops used in BR don't support all the features used by FIO yet and also there is a growing need for a test Infrastructure.

Strategy

Since NDMP dump/restore worked in 7G, a good idea of how it worked would give us insight on the changes that need to be made for BR. The difficulty lies in the improper documentation of FIO.
My first step is to figure out how FIO works on 7G and develop use cases to improve understanding.
Later I plan on extending the use cases to come up with a Unit Test for FIO.

Approach

1) The first thing that needs to be done is to Use NDMP dump/restore commands on Different filers.

2) Cover the NDMPcopy tests being done by the NDMP QA team.

List of commands can be found here.NDMP Services

3)The different data sets that will be made use for running these test commands are: (courtesy vijay)

   * Longlinks
   * nfsv4acls
   * ntacls
   * ntstreams
   * compat_data_dump (the full dataset used by NDMP QA).

4)The different scenarios for testing are:

   * IC to BR 7-mode
   * BR 7-mode to BR 10-mode
   * BR 7-mode to BR 7-mode
   * BR 7-mode to IC
   * BR 10-mode to BR-7 mode (cannot run treecompare)
   * BR 10-mode to BR 10-mode (cannot run treecompare)

5)This would give a good idea about how NDMP makes FIO calls.

6)Once this is done, using cscope various function calls made by NDMP needs to be analyzed. This would help in testing it functionally.

7)Results

**Consolidated test result**
Security Style	MIXED	NTFS
NT ACLs	Fail	Fail
NFS V4 ACLs	Fail	Fail
NT Streams	Fail	Fail
Sym Links	Pass	Fail
DOSbit	Fail	Fail
UID,GID,Permissions	Fail	Fail
mtime	Fail	Fail

Preparation for testing

6/27/08

Insert print statements for the commands in bkp_rcv_fio.c and run the 'ndmpcopy' command. This would give a clear idea about the order of execution of the functions.

Introduce break points in the code and run the code in the debugger mode. If the above method works, this one is probably not needed.

07/01/08

Results :

vidhyan> ndmpcopy -l 0 -st text -dt text -da root: /vol/newvol 10.97.129.201:/vol/newvol1

Ndmpcopy: Starting copy [ 8 ] ...

Ndmpcopy: vidhyan: Notify: Connection established

Ndmpcopy: 10.97.129.201: Notify: Connection established

Ndmpcopy: vidhyan: Connect: Authentication successful

Ndmpcopy: 10.97.129.201: Connect: Authentication successful

Ndmpcopy: vidhyan: Log: DUMP: creating "/vol/newvol/../snapshot_for_backup.0" snapshot.

Ndmpcopy: vidhyan: Log: DUMP: Using Full Volume Dump

fio_get_volume_from_path

fio_exist_metadir

fio_mkdir_metadir

fio_unlink_metadir

fio_mapfile_metadir

fio_vm_setsize

Ndmpcopy: vidhyan: Log: DUMP: Date of this level 0 dump: Tue Jul 8 16:48:41 2008.

Ndmpcopy: vidhyan: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: vidhyan: Log: DUMP: Dumping /vol/newvol to NDMP connection

Ndmpcopy: vidhyan: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: vidhyan: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: vidhyan: Log: DUMP: estimated 0 KB.

Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass V) [ACLs]

fio_unlink_metadir

fio_mapfile_metadir

fio_vm_setsize

fio_unlink_metadir

fio_mapfile_metadir

fio_vm_setsize

fio_statfs

fio_unlink_metadir

fio_creat_metadir

fio_unlink_metadir

fio_mapfile_metadir

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:12 2008: Begin level 0 restore

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:12 2008: Reading directories from the backup fio_vm_setsize

fio_write

fio_lseek

fio_close

fio_getfh

fio_unlink_metadir

fio_creat_metadir

fio_unlink_metadir

fio_mapfile_metadir

fio_vm_setsize

fio_open_metadir

fio_lseek

fio_read

fio_setattr

fio_qtree_enable_oplocks

fio_close

fio_open_metadir

fio_close

fio_creat

fio_write

fio_lseek

fio_write

fio_close

fio_setattr

fio_qtree_enable_oplocks

fio_close

fio_open_metadir

fio_open

fio_read

fio_lseek

fio_write

fio_close

fio_setattr

fio_qtree_enable_oplocks

fio_unlink_metadir

fio_unmapfile

fio_unlink_metadir

fio_unmapfile

fio_unlink_metadir

fio_unmapfile

fio_unlink_metadir

fio_unmapfile

fio_unlink_metadir

fio_close

fio_unlink_metadir

fio_unmapfile

fio_unlink_metadir

fio_exist_metadir

fio_recrmdir_metadir

Ndmpcopy: vidhyan: Log: DUMP: 11 KB

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Creating files and directories.

Ndmpcopy: vidhyan: Log: DUMP: DUMP IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Writing data to files.

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Restoring NT ACLs.

Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/newvol1/

Ndmpcopy: 10.97.129.201: Notify: restore successful

Ndmpcopy: vidhyan: Log: DUMP: Deleting "/vol/newvol/../snapshot_for_backup.0" snapshot.

Ndmpcopy: vidhyan: Notify: dump successful

Ndmpcopy: Transfer successful [ 47 seconds ]

Ndmpcopy: Done

NDMP copy and Treecompare result on the datasets

vidhyan-vsim1> ndmpcopy -l 0 -st text -dt text -da root: /vol/newvol 10.97.129.201:/vol/vidsvol

Ndmpcopy: Starting copy [ 1 ] ...

Ndmpcopy: vidhyan-vsim1: Notify: Connection established

Ndmpcopy: 10.97.129.201: Notify: Connection established

Ndmpcopy: vidhyan-vsim1: Connect: Authentication successful

Ndmpcopy: 10.97.129.201: Connect: Authentication successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: creating "/vol/newvol/../snapshot_for_backup.0" snapshot.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Using Full Volume Dump

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of this level 0 dump: Tue Jul 15 16:15:13 2008.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Dumping /vol/newvol to NDMP connection

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: estimated 0 KB.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass V) [ACLs]

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 16:15:48 2008: Begin level 0 restore

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 16:15:49 2008: Reading directories from the backup

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 16:15:56 2008: Creating files and directories.

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 16:15:56 2008: Writing data to files.

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 16:15:56 2008: Restoring NT ACLs.

Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: vidhyan-vsim1: Log: DUMP: 11 KB

Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/vidsvol/

Ndmpcopy: vidhyan-vsim1: Log: DUMP: DUMP IS DONE

Ndmpcopy: 10.97.129.201: Notify: restore successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Deleting "/vol/newvol/../snapshot_for_backup.0" snapshot.

Ndmpcopy: vidhyan-vsim1: Notify: dump successful

Ndmpcopy: Transfer successful [ 47 seconds ]

Ndmpcopy: Done

</nowiki>

treecompare output:

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1013

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1014

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1015

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1016

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1017

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1018

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1019

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1020

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1021

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1022

TREECMP: Found atime mismatch for /vol/vol2/.snapshot/snapvol2/1023

TREECMP: Found tree2 only directory entry entry ./rstabRENAME487CE727

TREECMP: Comparing special files.

TREECMP: Comparing extended attributes for directories.

TREECMP: Comparing files.

TREECMP: Found data mismatch: block 0 in ./restore_symboltable

TREECMP: Found data mismatch: block 1 in ./restore_symboltable

TREECMP: Found data mismatch: block 2 in ./restore_symboltable

TREECMP: Found data mismatch: block 3 in ./restore_symboltable

TREECMP: Found data mismatch: block 4 in ./restore_symboltable

TREECMP: Found data mismatch: block 5 in ./restore_symboltable

TREECMP: Found data mismatch: block 6 in ./restore_symboltable

TREECMP: Found data mismatch: block 7 in ./restore_symboltable

TREECMP: Found data mismatch: block 8 in ./restore_symboltable

TREECMP: Found data mismatch: block 9 in ./restore_symboltable

TREECMP: Found size mismatch for ./restore_symboltable

TREECMP: Found atime mismatch for ./restore_symboltable

TREECMP: Found mtime mismatch for ./restore_symboltable

TREECMP: Found crtime mismatch for ./restore_symboltable

TREECMP: Done. Wed Jul 16 17:10:41 GMT

tree1_only = 0, tree2_only = 1, mismatch = 1038

NFSv4 ACLS

n-vsim1> ndmpcopy -l 0 -st text -dt text -da r oot: /vol/vol3 10.97.129.201:/vol/vol4

Ndmpcopy: Starting copy [ 1 ] ...

Ndmpcopy: vidhyan-vsim1: Notify: Connection establis hed

Ndmpcopy: 10.97.129.201: Notify: Connection establis hed

Ndmpcopy: vidhyan-vsim1: Connect: Authentication suc cessful

Ndmpcopy: 10.97.129.201: Connect: Authentication suc cessful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: creating "/vol/vol3/../snapshot_for_backup.4" snapshot.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Using Full Volume Dump

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of this level 0 dump: Tue Jul 15 17:34:00 2008.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Dumping /vol/vol3 to NDMP connection

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: estimated 94 KB.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass V) [ACLs]

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:34:24 2008: Begin level 0 restore

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:34:24 2008: Reading directories from the backup

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:34:31 2008: Creating files and directories.

Ndmpcopy: 10.97.129.201: Log: RESTORE: The directory: ./restore_symboltable could not be restored with its original name.

Ndmpcopy: 10.97.129.201: Log: RESTORE: It has been renamed to: ./rstabRENAME487CE1C7

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:34:31 2008: Writing data to files.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: 93 KB

Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: vidhyan-vsim1: Log: DUMP: DUMP IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/vol4/

Ndmpcopy: 10.97.129.201: Notify: restore successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Deleting "/vol/vol3/../snapshot_for_backup.4" snapshot.

Ndmpcopy: vidhyan-vsim1: Notify: dump successful

Ndmpcopy: Transfer successful [ 38 seconds ]

Ndmpcopy: Done

Treecompare output: vidhyan-vsim1*> treecompare start -c -L checksum /vol/vol3/.snapshot/snapvol3 /vol/vol4/.snapshot/snapvol4

TREECMP: Start. Wed Jul 16 17:06:34 GMT, compare with checksum

TREECMP: Comparing directories.

TREECMP: Found atime mismatch for .

TREECMP: Found tree2 only directory entry entry ./rstabRENAME487CE1C7

TREECMP: Comparing special files.

TREECMP: Comparing extended attributes for directories.

TREECMP: Comparing files.

TREECMP: Found data mismatch: block 0 in ./restore_symboltable

TREECMP: Found data mismatch: block 1 in ./restore_symboltable

TREECMP: Found data mismatch: block 2 in ./restore_symboltable

TREECMP: Found data mismatch: block 3 in ./restore_symboltable

TREECMP: Found data mismatch: block 4 in ./restore_symboltable

TREECMP: Found data mismatch: block 5 in ./restore_symboltable

TREECMP: Found size mismatch for ./restore_symboltable

TREECMP: Found atime mismatch for ./restore_symboltable

TREECMP: Found mtime mismatch for ./restore_symboltable

TREECMP: Found crtime mismatch for ./restore_symboltable

TREECMP: Done. Wed Jul 16 17:06:34 GMT

tree1_only = 0, tree2_only = 1, mismatch = 11

NTACLS

n-vsim1> ndmpcopy -l 0 -st text -dt text -da r oot: /vol/vol5 10.97.129.201:/vol/vol6

Ndmpcopy: Starting copy [ 2 ] ...

Ndmpcopy: vidhyan-vsim1: Notify: Connection establis hed

Ndmpcopy: 10.97.129.201: Notify: Connection establis hed

Ndmpcopy: vidhyan-vsim1: Connect: Authentication suc cessful

Ndmpcopy: 10.97.129.201: Connect: Authentication suc cessful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: creating "/vol/v ol5/../snapshot_for_backup.7" snapshot.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Using Full Volum e Dump

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of this level 0 dump: Tue Jul 15 17:40:14 2008.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Dumping /vol/vol5 to NDMP connection

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: estimated 21571 KB.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:40:38 2008: Begin level 0 restore

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:40:38 2008: Reading directories from the backup

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:40:45 2008: Creating files and directories.

Ndmpcopy: 10.97.129.201: Log: RESTORE: The directory: ./restore_symboltable could not be restored with its original name.

Ndmpcopy: 10.97.129.201: Log: RESTORE: It has been renamed to: ./rstabRENAME487CF84BNdmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:40:45 2008: Writing data to files.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass V) [ACLs]

Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/vol6/

Ndmpcopy: 10.97.129.201: Notify: restore successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: 21575 KB

Ndmpcopy: vidhyan-vsim1: Log: DUMP: DUMP IS DONE

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Deleting "/vol/vol5/../snapshot_for_backup.7" snapshot.

Ndmpcopy: vidhyan-vsim1: Notify: dump successful

Ndmpcopy: Transfer successful [ 45 seconds ]

Ndmpcopy: Done

Treecompare output:

vidhyan-vsim1*> treecompare start -c -L checksum /vol/vol5/.snapshot/snapvol5 /vol/vol6/.snapshot/snapvol6

TREECMP: Start. Wed Jul 16 17:14:48 GMT, compare with checksum

TREECMP: Comparing directories.

TREECMP: Found atime mismatch for .

TREECMP: Found tree2 only directory entry entry ./rstabRENAME487CF84B

TREECMP: Comparing special files.

TREECMP: Comparing extended attributes for directories.

TREECMP: Comparing files.

TREECMP: Found data mismatch: block 0 in ./restore_symboltable

TREECMP: Found data mismatch: block 1 in ./restore_symboltable

TREECMP: Found data mismatch: block 2 in ./restore_symboltable

TREECMP: Found data mismatch: block 3 in ./restore_symboltable

TREECMP: Found data mismatch: block 4 in ./restore_symboltable

TREECMP: Found data mismatch: block 5 in ./restore_symboltable

TREECMP: Found size mismatch for ./restore_symboltable

TREECMP: Found atime mismatch for ./restore_symboltable

TREECMP: Found mtime mismatch for ./restore_symboltable

TREECMP: Found crtime mismatch for ./restore_symboltable

TREECMP: Done. Wed Jul 16 17:15:41 GMT

tree1_only = 0, tree2_only = 1, mismatch = 11

   * NTStreams

n-vsim1> ndmpcopy -l 0 -st text -dt text -da root: /vol/vol7 10.97.129.201:/vo l/vol8

Ndmpcopy: Starting copy [ 3 ] ...

Ndmpcopy: vidhyan-vsim1: Notify: Connection established

Ndmpcopy: 10.97.129.201: Notify: Connection established

Ndmpcopy: vidhyan-vsim1: Connect: Authentication successful

Ndmpcopy: 10.97.129.201: Connect: Authentication successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: creating "/vol/vol7/../snapshot_for_backup.10" s napshot.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Using Full Volume Dump

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of this level 0 dump: Tue Jul 15 17:45:30 2 008.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Dumping /vol/vol7 to NDMP connection

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: estimated 359 KB.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: vidhyan-vsim1: Log: DUMP: dumping (Pass V) [ACLs]

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:45:56 2008: Begin level 0 resto re

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:45:56 2008: Reading directories from the backup

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:46:02 2008: Creating files and directories.

Ndmpcopy: 10.97.129.201: Log: RESTORE: The directory: ./restore_symboltable could no t be restored with its original name.

Ndmpcopy: 10.97.129.201: Log: RESTORE: It has been renamed to: ./rstabRENAME487CF99B

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:46:02 2008: Writing data to fil es.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: 305 KB

Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 15 17:46:03 2008: Restoring NT ACLs.

Ndmpcopy: vidhyan-vsim1: Log: DUMP: DUMP IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/vol8/

Ndmpcopy: 10.97.129.201: Notify: restore successful

Ndmpcopy: vidhyan-vsim1: Log: DUMP: Deleting "/vol/vol7/../snapshot_for_backup.10" s napshot.

Ndmpcopy: vidhyan-vsim1: Notify: dump successful

Ndmpcopy: Transfer successful [ 40 seconds ]

Ndmpcopy: Done

treecompare output:

vidhyan-vsim1*> treecompare start -c -L checksum /vol/vol7/.snapshot/snapvol7 /vol/vol8/.snapshot/snapvol8

TREECMP: Start. Wed Jul 16 17:17:12 GMT, compare with checksum

TREECMP: Comparing directories.

TREECMP: Found tree2 only directory entry entry ./rstabRENAME487CF99B

TREECMP: Comparing special files.

TREECMP: Comparing extended attributes for directories.

TREECMP: Comparing files.

TREECMP: Found data mismatch: block 0 in ./restore_symboltable

TREECMP: Found data mismatch: block 1 in ./restore_symboltable

TREECMP: Found data mismatch: block 2 in ./restore_symboltable

TREECMP: Found data mismatch: block 3 in ./restore_symboltable

TREECMP: Found data mismatch: block 4 in ./restore_symboltable

TREECMP: Found data mismatch: block 5 in ./restore_symboltable

TREECMP: Found size mismatch for ./restore_symboltable

TREECMP: Found atime mismatch for ./restore_symboltable

TREECMP: Found mtime mismatch for ./restore_symboltable

TREECMP: Found crtime mismatch for ./restore_symboltable

TREECMP: Done. Wed Jul 16 17:17:16 GMT

tree1_only = 0, tree2_only = 1, mismatch = 10

Problems faced

BEWARE OF OUTDATED WIKI's. THERE ARE TONS OF THEM!!
P4 License

Even after taking the quiz successfully, it did not show up in the stored result. Make sure 1. You send an email to dl-p4-admins after taking the quiz so that they know you took the quiz.

2. .b4p4rc file is correct.

Vsims

Had difficulty in cluster creation. The command given in wikid has some missing fields.

The correct cluster create command is :

vidhyan:vsim1::> cluster create -clustername <vsim name> -license NWUZZJPJYBFDAA -ipmgmt <cluster mgmt ip> -netmaskmgmt 255.255.248.0 -gatewaymgmt 10.97.0.1 -portmgmt e0d -ipaddr1 <c1 IP> -netmask1 255.255.248.0 -ipaddr2 <c2 IP> -netmask2 255.255.248.0

Useful Material

These links helped me understand WAFL, FIO and ONTAP.

WAFL&FIO

 http://wikid.netapp.com/w/User:Rsmith/WAFL_Intro
 http://wikid.netapp.com/w/Nane-Wafl:Main

ONTAP

 http://wikid.netapp.com/w/ConvergenceArchitectureSpec
 http://wikid.netapp.com/w/HHG/Software-4.0/ClusteredONTAPDevProcess
 http://busops-web.corp.netapp.com/core_systems/convergence/index.html

BR Spec Depot

 http://wikid.netapp.com/w/Boilermaker_Design_Depot

Coverity

 http://coverity.eng.netapp.com:5105/doc/checker_ref.html

Training Videos

 http://wikid.netapp.com/w/Nane-Wafl:Online_Training

Vsims

 http://wikid.netapp.com/w/OS_Group/Simulator/VSim_User_Guide

P4 guide

 http://wikid.netapp.com/w/HHG/Perforce_Primer-2.6/Quick_Start

Notes

NDMP dump/restore - How it works?

The NDMP dump/restore is basically a means of backing up data. In case something goes wrong with the current data, the dumped data can be restored and normal operations can be done on it as usual. Its a way of making a remote copy.

Steps followed :

1. A snapshot of the existing file system is created. A snapshot is a read only copy.

2. While issuing a dump command, The dump module reads the snapshot and figures out the hierarchy of the file system to be dumped.

3. This structure is first written to a serialized interface where the data is to be dumped. This structure format is used at the restore side to recreate the file hierarchy.

4. After dumping the hierarchy(empty nodes) the data is also dumped to the serialized interface.

5. While restoring, the restore module first reads the hierarchy of the file system ( since it was written first - FIFO) and created empty directories and files.

6. later it reads the data from the interface and restores it at the destination.

7. In order to restore the data back to the source, a dump has to be done to interface from the destination and then a restore has to be done to source.

Qtrees

Qtrees are special sub-directories in a volume that acts as a virtual subvolume with special attributes, primarily quotas and permissions. They are usually limited by size.

NDMPCOPY procedure:

1.Open a terminal. log into one of the cycl servers.(say cycl09)

2.Do a vsim make

  [vidhyan@cycl09] vsim make -quickl -ir /u/vidhyan/p4/new( client name) -d x86 -7m vsimnew( vsim identifier)

(remember to include -d for the debugger mode for 7m)

3.Once vsim make is done. connect to the server and create console

  [vidhyan@cycl09] vsim rcons vsimnew

  [vidhyan@cycl09] vsim cons vsimnew

The terminal would hang. Just leave that as it is and open another terminal

4. Start the Vsim

  [vidhyan@cycl09] vsim start vsimnew

5. The previous terminal where the rcons and cons commands were running would load the simulator. In case it goes to the boot menu select option 4 press ctrl + c after "7-mode system"

Make sure the client is updated. Else you might hit on Burt 299558 and encounter the following when you choose "Normal boot" form the menu.

disk (A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0) not found disk (C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F) not found disk (00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B) not found disk (A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0) not found disk (C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F) not found disk (00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B:00C0A82F:C887BB0B) not found disk (A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0:A82FC887:BB0B00C0) not found

6.If this happens, then choose option 44/7 from the 1-7 option and set disk own all.

7.If it goes to the SIMLOADER prompt,

  SIMLOADER> setup 
  Are you sure you want to continue? [yes] yes
  Please enter the new hostname [vidhyan-vsim1]:
  Do you want to configure interface groups? [n]:
  Please enter the IP address for Network Interface e0a []:
  Should interface e0a take over a partner IP address during failover? [n]:
  Please enter the IP address for Network Interface e0b []:
  Should interface e0b take over a partner IP address during failover? [n]:
  Please enter the IP address for Network Interface e0d [10.97.129.201]:
  Please enter the netmask for Network Interface e0d [255.0.0.0]: 255.255.248.0
  Should interface e0d take over a partner IP address during failover? [n]:
  Please enter media type for e0d {100tx-fd, tp-fd, 100tx, tp, auto (10/100/1000)} [auto]:
  Please enter flow control for e0d {none, receive, send, full} [full]:
  Do you want e0d to support jumbo frames? [n]:
  Would you like to continue setup through the web interface? [n]:
  Please enter the name or IP address of the default gateway [10.97.128.1]:
  Please enter the name or IP address of the administration host [10.97.129.203]:
  Please enter timezone [GMT]:
  Where is the filer located? [nane]:
  Do you want to run DNS resolver? [n]:
  Do you want to run NIS client? [n]:
  The initial aggregate currently contains 3 disks;  you may add more
  disks to it later using the "aggr add" command.

Now type 'reboot' for changes to take effect. vidhyan-vsim1> reboot kldunload: attempt to unload file that was loaded by the kernel kldunload: can't unload file: Device busy Terminated SK: exiting CPU 1 DBG: Shut down wafl_sync & network took 1847 msecs DBG: Shut down RAID hilevel took 0 msecs DBG: Shut down WAFL took 3416 msecs (0 0 0 770 704 719 1223) DBG: Shut down wafl took 3416 msecs DBG: Shut down coral took 0 msecs DBG: Shut down RAID took 340 msecs DBG: Shut down fm took 0 msecs DBG: Total time in shutdown_appliance: 6050 ms SK: exiting CPU 0 Uptime: 2h49m20s "vsimsetup" 266 lines, 17747 characters vidhyan-vsim1>

8. AGGREGATE CREATE COMMAND:

 vidhyan-vsim1> aggr create newaggr 6

Mon Jul 7 16:52:21 GMT [raid.fsm.commitStateTransit:debug]: /newaggr (VOL): raid state change UNINITD -> NEEDFS Mon Jul 7 16:52:21 GMT [raid.fsm.commitStateTransit:debug]: /newaggr (MIRROR): raid state change UNINITD -> UNMIRRORED Mon Jul 7 16:52:21 GMT [raid.fsm.commitStateTransit:debug]: /newaggr/plex0 (PLEX): raid state change UNINITD -> EMPTY Mon Jul 7 16:52:21 GMT [raid.fsm.commitStateTransit:debug]: /newaggr/plex0 (PLEX): raid state change EMPTY -> NORMAL Mon Jul 7 16:52:21 GMT [raid.fsm.commitStateTransit:debug]: /newaggr/plex0/rg0 (GROUP): raid state change UNINITD -> NORMAL Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v4.20 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [12052504] to aggregate newaggr has completed successfully Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v5.19 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [14292603] to aggregate newaggr has completed successfully Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v4.19 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [12052503] to aggregate newaggr has completed successfully Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v5.18 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [14292602] to aggregate newaggr has completed successfully Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v4.18 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [12052502] to aggregate newaggr has completed successfully Mon Jul 7 16:52:21 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /newaggr/plex0/rg0/v5.17 Shelf ? Bay ? [NETAPP VD-1000MB-FZ 0042] S/N [14292601] to aggregate newaggr has completed successfully Creation of an aggregate with 6 disks has completed.

9.VOL CREATION:

 vidhyan-vsim1> vol create newvol newaggr 200m

DBG: Assigning uuid '3726904e-4dd4-11dd-914b-000c299e1213' to new flexible volume 'newvol' (FSID 0x13c1c26) Mon Jul 7 17:00:04 GMT [kern.uptime.filer:info]: 5:00pm up 33 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops Mon Jul 7 17:00:04 GMT [monitor.shelf.accessError:CRITICAL]: Enclosure services has detected an error in access to shelves on channel v0. Mon Jul 7 17:00:04 GMT [monitor.shelf.accessError:CRITICAL]: Enclosure services has detected an error in access to shelves on channel v1. exportfs [Line 2]: Export of local volume /vol/vol0 requires nfs license exportfs [Line 3]: Export of local volume /vol/vol0/home requires nfs license exportfs [Line 2]: Export of local volume /vol/newvol requires nfs license Creation of volume 'newvol' with size 200m on containing aggregate 'newaggr' has completed.

10.COPY DATASET TO THE VOLUME:

 [vidhyan@cycl09 vsrinath]$ gunzip -c /u/vsrinath/fio/compat_data.dmp.gz | rsh -l root vidhyan-vsim1 "restore rfD - /vol/newvol"

11.NDMPCOPY :

 vidhyan> options ndmpd.authtype plaintext

Tue Jul 8 16:48:19 GMT [kern.cli.cmd:debug]: Command line input: the command is 'options'. The full command line is 'options ndmpd.authtype plaintext'.

 vidhyan> options ndmpd.authtype
 ndmpd.authtype               plaintext

 vidhyan> ndmpcopy -l 0 -st text -dt text -da root:  /vol/newvol 10.97.129.201:/vol/newvol1

Ndmpcopy: Starting copy [ 8 ] ... Ndmpcopy: vidhyan: Notify: Connection established Ndmpcopy: 10.97.129.201: Notify: Connection established Ndmpcopy: vidhyan: Connect: Authentication successful Ndmpcopy: 10.97.129.201: Connect: Authentication successful Ndmpcopy: vidhyan: Log: DUMP: creating "/vol/newvol/../snapshot_for_backup.0" snapshot. Ndmpcopy: vidhyan: Log: DUMP: Using Full Volume Dump fio_get_volume_from_path fio_exist_metadir fio_mkdir_metadir fio_mkdir_metadir fio_unlink_metadir fio_mapfile_metadir fio_vm_setsize Ndmpcopy: vidhyan: Log: DUMP: Date of this level 0 dump: Tue Jul 8 16:48:41 2008. Ndmpcopy: vidhyan: Log: DUMP: Date of last level 0 dump: the epoch. Ndmpcopy: vidhyan: Log: DUMP: Dumping /vol/newvol to NDMP connection Ndmpcopy: vidhyan: Log: DUMP: mapping (Pass I)[regular files] Ndmpcopy: vidhyan: Log: DUMP: mapping (Pass II)[directories] Ndmpcopy: vidhyan: Log: DUMP: estimated 0 KB. Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass III) [directories] Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass IV) [regular files] Ndmpcopy: vidhyan: Log: DUMP: dumping (Pass V) [ACLs] fio_unlink_metadir fio_mapfile_metadir fio_vm_setsize fio_unlink_metadir fio_mapfile_metadir fio_vm_setsize fio_statfs fio_unlink_metadir fio_creat_metadir fio_unlink_metadir fio_mapfile_metadir Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:12 2008: Begin level 0 restore Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:12 2008: Reading directories from the backup fio_vm_setsize fio_write fio_lseek fio_close fio_getfh fio_unlink_metadir fio_creat_metadir fio_unlink_metadir fio_mapfile_metadir fio_vm_setsize fio_open_metadir fio_lseek fio_read fio_setattr fio_qtree_enable_oplocks fio_close fio_open_metadir fio_close fio_creat fio_write fio_lseek fio_write fio_write fio_write fio_close fio_setattr fio_qtree_enable_oplocks fio_close fio_open_metadir fio_open fio_read fio_lseek fio_write fio_close fio_setattr fio_qtree_enable_oplocks fio_unlink_metadir fio_unmapfile fio_unlink_metadir fio_unlink_metadir fio_unlink_metadir fio_unmapfile fio_unlink_metadir fio_unmapfile fio_unlink_metadir fio_unmapfile fio_unlink_metadir fio_close fio_unlink_metadir fio_unmapfile fio_unlink_metadir fio_exist_metadir fio_recrmdir_metadir Ndmpcopy: vidhyan: Log: DUMP: 11 KB Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Creating files and directories. Ndmpcopy: vidhyan: Log: DUMP: DUMP IS DONE Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Writing data to files. Ndmpcopy: 10.97.129.201: Log: RESTORE: Tue Jul 8 16:49:19 2008: Restoring NT ACLs. Ndmpcopy: 10.97.129.201: Log: RESTORE: RESTORE IS DONE Ndmpcopy: 10.97.129.201: Log: RESTORE: The destination path is /vol/newvol1/ Ndmpcopy: 10.97.129.201: Notify: restore successful Ndmpcopy: vidhyan: Log: DUMP: Deleting "/vol/newvol/../snapshot_for_backup.0" snapshot. Ndmpcopy: vidhyan: Notify: dump successful Ndmpcopy: Transfer successful [ 47 seconds ] Ndmpcopy: Done

11. MOUNT THE EXISTING VOLUME TO A DIRECTORY IN THE SIMULATOR TO EXPLORE ITS CONTENTS.

 [vidhyan@siml2 ~]$ sudo sim_mount vidhyan-vsim1 newvol2

+ mount -o rw,bg,hard,intr,rsize=8192,wsize=8192 vidhyan-vsim1:/vol/newvol2 /t /vidhyan-vsim1/newvol2 [vidhyan@siml2 ~]$

Accessing the mounted data in the simulator directory :

[vidhyan@siml2 ~]$ cd /t/vidhyan-vsim1 [vidhyan@siml2 vidhyan-vsim1]$ ls newvol2 u [vidhyan@siml2 vidhyan-vsim1]$ cd newvol2 [vidhyan@siml2 newvol2]$ ls batch compat_unix README sid compat_mixed etc restore_symboltable vol_log.zip compat_ntfs fscreator rstabRENAME41FADC18

Notes

login to one of the linux simulators:

[vidhyan@cycl09~]ssh siml1.nane.netapp.com

[vidhyan@siml1 ~]cd /u/vidhyan/p4

[vidhyan@siml1 ~]

remem:

[vidhyan@cycl09] options

options rsh on

TEST RESULTS:

vidhyan-vsim1> ndmpcopy -l 0 -st text -dt text -da root: /vol/newvol vidhyan-vsim1:/vol/newvol1

Ndmpcopy: vidhyan-vsim1: Notify: Connection Established

license add UYTCNCG

Starting applications specified in /u/vidhyan/.vnc/xstartup Log file is /u/vidhyan/.vnc/cycl09.nane.netapp.com:62.log

[vidhyan@cycl09 ~]$ vncserver :62 -geometry 1400x1050

New 'cycl09.nane.netapp.com:62 (vidhyan)' desktop is cycl09.nane.netapp.com:62

license

Service License Code Mnemonic

cifs               GTLYRVH  Gaudier Than Last Year's Restored Victorian Home  
cluster            KDQLGBN  Kill Dan Quayle, Leave George Bush Neutered  
cluster_remote     GTFJHSI  George, The Forgetful Just Hate Special Instructions  
dafs               OBOLDQC  Obey Bush's Orders, Lest Dan Quayle Cry  
disk_sanitization  CPUPBIL     
fcp                ICRNCZG  Incoherent Comments Really Need Calm Zen Grading  
flex_cache         KRFUWOJ   (good up to and including 7.2x)
flexcache_nfs      OSRAREM
flex_clone         QNFYRVH     
http               AODUWOJ  America Online Defames Usenet With Obnoxious Juveniles  
iscsi              IKVAREM  I Keep Various Averices Repressed, Ere Manifesting  
nfs                UYTCNCG  Unless You Take Control, Nasty Code Grows  
smdomino           CFZRXFF  Can Fearsome Zealots Read Xerox Formatted Files  
smsql              SHEWSMD  Sigh Heavily Even When Someone's Making Dinner  
snaplock (pre 7.0.0.1)        IASLGBN  I Abhor Sloppy Little Gnomes Being Neat  
snaplock (7.0.0.1 and above)  KJHHIJE   
snaplock_enterprise           AOXRXFF  Another Obnoxious Xenophobe Retards Xerox Flame Fest  
snapmanagerexchange           GFMYRVH  Gophers Filched My Yearly Ripe Vegetable Harvest  
snapmirror         UUNLDQC  Unusual Unix Nerds Love Dan Quayle's Cooking  
snapmirror_sync    IEOAOTB  I Eat Only After Others Take Baths  
snapmover          KBNNOFF  Keep Bill's Naughty Naked Outings From Family  
snaprestore        EOYPBIL  Eunice, Only Your Peanut Butter Is Left  
snapvalidator      MCGWSMD     
sv_linux_pri       GROPYWA  Give Readily, Or Put Your Wampum Away  
sv_ontap_pri       YAYCNCG  Yodeling Affects Your Concentration Near Comely Girls  
sv_ontap_sec       KVPYRVH  Kindness Vectors Peace, Yielding Resilient Virtuous Habits  
sv_unix_pri        IUILGBN  It's Understandable If Logs Grow Bigger Needlessly  
sv_windows_pri     SLGLDQC  Stop Letting George Lose Dan Quale's Car  
syncmirror_local   UELJHSI  Understand, Everyone. Lincoln Just Has Some Issues  
vfiler             SPQLGBN  Sam, Please Quit Letting George Bush Negotiate  
vld                QYOAREM  Quality Yes, Or All Remarkable Errors Manifest  
volcopy            AAWPYWA  Angry Aardvarks Will Pelt You With Acorns (obsolete, but in case you have an old platform)

The following are codes for promoted (production) simulators

Service             License Code  
cifs                DZDACHD     
cluster             JQIGWWF     
cluster_remote      BCZZBHD     
dafs                TBPGWWF     
disk_sanitization   PZKEAZL     
fcp                 BKHEXNB     
flex_cache          HFAYFJJ     
flex_clone          ANLEAZL     
http                NAZOMKC      
http_compression    SEDAFSN     
iscsi               BSLRLTG     
rapid_restore       CMRGWWF
nearstore_option    ELNRLTG
nfs                 BQOEAZL     
smdomino            RKBAFSN     
smsql               HNGEAZL     
snaplock (pre 7.0.0.1)        BIICBQH     
snaplock (7.0.0.1 and above)  ZOJPPVM     
snapmanagerexchange BCJEAZL     
snapmirror          DFVXFJJ     
snapmirror_sync     XJQIVFK
snaprestore         DNDCBQH     
snapvalidator       JQAACHD     
socks               UXBNQMI     
sv_linux_pri        ZYICXLC     
sv_ontap_pri        PVOIVFK     
sv_ontap_sec        PDXMQMI     
sv_unix_pri         RQAYBFE     
sv_windows_pri      ZOPRKAM     
syncmirror_local    RIQTKCL     
vfiler              NQBYFJJ     
vld                 JGFRLTG

NDMP COPY outputs