Category Archives: Uncategorized

Reusing a disk in Storage Spaces

When trying to reuse a disk that was previously part of a storage space I find using get-physicaldisk that the disk cannot be part of a pool.  Use reset-physicaldisk -uniqueid to reset the disk.

Command line create partition and format in Linux with a label

I have a Raspberry Pi with a couple of memory sticks (8/32GB SanDisk). I’d like to plug them into my Raspberry Pi, replace the partitions and format them to ext4 with a label describing the partition.

A note before I begin is that you will need to run most of the command with elevated privileges (root) so I’m prefixing the required command with sudo.

Using lsblk –fs you will be able to identify the physical drives (devices) in your system as well as any partitions they have on them.

pi@raspberrypi:~ $ sudo lsblk --fs
NAME        FSTYPE LABEL UUID                                 MOUNTPOINT
sda                                                           
└─sda1      vfat         FDB1-8F68                            
sdb                                                           
└─sdb1      vfat         2BB1-F497                            /media/pi/2BB1-F497
mmcblk0                                                       
├─mmcblk0p1 vfat   boot  7EF5-979D                            /boot
└─mmcblk0p2 ext4         139ca52c-d45e-44ca-8b8c-12c0af0c9cf5 /

Firstly ensure that any partition(s) on the drive(s) you want to work on are unmounted.

pi@raspberrypi:~ $ sudo umount /dev/sda1
pi@raspberrypi:~ $ sudo umount /dev/sdb1

Launch parted. On launch you will be shown the version number of parted as well as the device you are currently using.  You will then be left at a prompt  that reads (parted).

pi@raspberrypi:~ $ sudo parted 
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)

In parted you can type help to bring up the help system overview.

(parted) help                                                             
  align-check TYPE N                        check partition N for TYPE(min|opt) alignment
  help [COMMAND]                           print general help, or help on COMMAND
  mklabel,mktable LABEL-TYPE               create a new disklabel (partition table)
  mkpart PART-TYPE [FS-TYPE] START END     make a partition
  name NUMBER NAME                         name partition NUMBER as NAME
  print [devices|free|list,all|NUMBER]     display the partition table, available devices, free space, all found
        partitions, or a particular partition
  quit                                     exit program
  rescue START END                         rescue a lost partition near START and END
  resizepart NUMBER END                    resize partition NUMBER
  rm NUMBER                                delete partition NUMBER
  select DEVICE                            choose the device to edit
  disk_set FLAG STATE                      change the FLAG on selected device
  disk_toggle [FLAG]                       toggle the state of FLAG on selected device
  set NUMBER FLAG STATE                    change the FLAG on partition NUMBER
  toggle [NUMBER [FLAG]]                   toggle the state of FLAG on partition NUMBER
  unit UNIT                                set the default unit to UNIT
  version                                  display the version number and copyright information of GNU Parted

For more detailed help on a specific command type help followed by the command you want help with.

(parted) help print                                                       
  print [devices|free|list,all|NUMBER]     display the partition table, available devices, free space, all found
        partitions, or a particular partition

        Without arguments, 'print' displays the entire partition table. However with the following arguments it performs
        various other actions.
          devices   : display all active block devices
          free      : display information about free unpartitioned space on the current block device
          list, all : display the partition tables of all active block devices
          NUMBER    : display more detailed information about this particular partition

You can get a list of all devices and their partitions by entering the print all command. As you can see from the listing below the Raspberry Pi boots from an SD card so it’s a device called mmcblk0. On most computers the device that has the boot partition on it will be sda. As sda has not already been used sda has been issued to one memory stick and sdb has been issued to the other. sda and sdb are both currently fat32 and part of the goal here is to format them as ext4.

(parted) print all                                                        
Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sda: 32.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  32.0GB  32.0GB  fat32              msftdata

Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sdb: 8004MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  8004MB  8003MB  fat32              msftdata

Model: SD SE64G (sd/mmc)
Disk /dev/mmcblk0: 63.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  70.3MB  66.1MB  primary  fat32        lba
 2      70.3MB  63.9GB  63.8GB  primary  ext4

Even though we were told that /dev/sda was the device we were using when we opened parted you can confirm or change the device you’re using with the following command.

(parted) select /dev/sda
Using /dev/sda

Next we’re going to remove the partitions from sda and sdb so let’s look up the help for the rm command.

(parted) help rm                                                          
  rm NUMBER                                delete partition NUMBER

        NUMBER is the partition number used by Linux.  On MS-DOS disk labels, the primary partitions number from 1 to 4,
        logical partitions from 5 onwards.

Since we are using the device sda and sda only has one partition (the partition number is 1) the command to remove that partition is quite short.

If you get a message saying that the device is being used then the drive will need unmounting outside of parted using umount (see above).

(parted) rm 1

If you enter the print command without any further parameters the information returned with be for the current device. In the case of the example below it is for the device /dev/sda and shows that partition 1 has now been removed.

(parted) print                                                            
Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sda: 32.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags

Now let’s switch to the device sdb.

(parted) select /dev/sdb
Using /dev/sdb

From the print all output above we can see that there is also only one partition on sdb and that it is also partition 1 so we can remove this partition like we removed the partition on sda.

(parted) rm 1

To check that the partition was removed successfully we can again using the print command.

(parted) print
Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sdb: 8004MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags

Next we’re going to make a partition table so let’s look up the help for that.

(parted) help mktable                                                     
  mklabel,mktable LABEL-TYPE               create a new disklabel (partition table)

        LABEL-TYPE is one of: aix, amiga, bsd, dvh, gpt, mac, msdos, pc98, sun, loop

If you type mktable without any parameters you enter interactive mode and will be prompted for the inputs required.

(parted) mktable 
New disk label type? gpt
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to
continue?
Yes/No? yes

Next let’s add a partition. The partition will be a primary partition using the file system type ext4. The partition will start after 1MB and use 100% of the rest of the device.

(parted) mkpart                                                           
Partition name?  []? primary                                              
File system type?  [ext2]? ext4
Start? 1MB
End? 100%

If you use print now you’ll be able to see the new partition has been created.

(parted) print                                                            
Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sda: 32.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  32.0GB  32.0GB  ext4         primary

Now we do the same for sdb, let’s switch to sdb.

(parted) select /dev/sdb
Using /dev/sdb

Make the partition table.

(parted) mktable                                                          
New disk label type? gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to
continue?
Yes/No? yes

Now let’s make the next partition using the same settings as with sda.

(parted) mkpart 
Partition name?  []? primary
File system type?  [ext2]? ext4
Start? 1MB                                                                
End? 100%

Now using print we can confirm that the partition has been made.

(parted) print                                                            
Model: SanDisk Cruzer Blade (scsi)
Disk /dev/sdb: 8004MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  8004MB  8003MB  ext4         primary

Entering the command q will quit parted. There is no need to save your changes as they are saved as you go.

(parted) q                                                                
Information: You may need to update /etc/fstab.

Now we want to format the ext4 file system onto the partition, we will use mkfs.ext4. You can look up the help for this using the man command.

pi@raspberrypi:~ $ man mkfs.ext4

The command starts with the -L parameter to specify the partitions label, next you pass the device followed immediately by the partition number.

pi@raspberrypi:~ $ sudo mkfs.ext4 -L SanDisk_32GB /dev/sda1
mke2fs 1.42.12 (29-Aug-2014)
/dev/sda1 contains a vfat file system
Proceed anyway? (y,n) y
Creating filesystem with 7815936 4k blocks and 1954064 inodes
Filesystem UUID: 6661f044-c533-4eb0-9a23-89aa6356a8a3
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Let’s do the same for sdb.

pi@raspberrypi:~ $ sudo mkfs.ext4 -L SanDisk_8GB /dev/sdb1
mke2fs 1.42.12 (29-Aug-2014)
/dev/sdb1 contains a vfat file system
Proceed anyway? (y,n) y
Creating filesystem with 1953792 4k blocks and 488640 inodes
Filesystem UUID: 3f51678b-b2b6-465a-b464-1e461058b0b9
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Finally we can use lsblk –fs to see the new partitions with their file systems, labels and UUID’s. If you were to eject and reinsert the USB device or restart the machine they would mount themselves.

pi@raspberrypi:~ $ sudo lsblk --fs
NAME        FSTYPE LABEL        UUID                                 MOUNTPOINT
sda                                                                  
└─sda1      ext4   SanDisk_32GB 6661f044-c533-4eb0-9a23-89aa6356a8a3 
sdb                                                                  
└─sdb1      ext4   SanDisk_8GB  3f51678b-b2b6-465a-b464-1e461058b0b9 
mmcblk0                                                              
├─mmcblk0p1 vfat   boot         7EF5-979D                            /boot
└─mmcblk0p2 ext4                139ca52c-d45e-44ca-8b8c-12c0af0c9cf5 /

Restarting the machine.

pi@raspberrypi:~ $ sudo init 6

After a reboot you can now see the USB devices mounted.

pi@raspberrypi:~ $ sudo lsblk --fs
NAME        FSTYPE LABEL        UUID                                 MOUNTPOINT
sda                                                                  
└─sda1      ext4   SanDisk_32GB 6661f044-c533-4eb0-9a23-89aa6356a8a3 /media/pi/SanDisk_32GB
sdb                                                                  
└─sdb1      ext4   SanDisk_8GB  3f51678b-b2b6-465a-b464-1e461058b0b9 /media/pi/SanDisk_8GB
mmcblk0                                                              
├─mmcblk0p1 vfat   boot         7EF5-979D                            /boot
└─mmcblk0p2 ext4                139ca52c-d45e-44ca-8b8c-12c0af0c9cf5 /

RDiff-Backup from a Windows machine to Linux

I wanted to backup user profiles a number of workstations onto a Linux machine.  The workstation are Windows machines and I’ve round RDiff-Backup to be a very robust and efficient too.  RDiff-Backup works much like RSync but where RSync is useful for making mirrors of directories RDiff-Backup will also give you generation of those mirrors to refer to.

One of my prerequisites for the backup is for there to be a minimal amount of configuration to need to be done to the workstations in order for them to backup their profile.  I don’t want to install any software or change any PATH variables.

What I ended up doing was downloading the Windows version of RDiff-Backup from here as well as the required Visual Studio files from here and finally a copy of Putty’s plink.exe from here.  I then saved/extracted all three into a folder in the “Program Files” directory called “rdiff-backup-1.2.8”.

Next I generated an RSA key pair using instructions from here.  And copied the id_rsa.pub to my Windows machine.  Using puttygen.exe which I downloaded from here I followed these instructions to turn the public half of my key into a .ppk file (without a passphrase so it would work in the script below).

I saved the .ppk file into the same directory as I’d saved the other files and ran the command below.


"C:\Program Files\rdiff-backup-1.2.8\rdiff-backup.exe" -v5 --print-statistics --exclude-symbolic-links --no-hard-links --exclude "C:/Users/smagee/AppData" --exclude "C:/Users/smagee/SMAGEE" --exclude "C:/Users/smagee/OneDrive" --exclude "C:/Users/smagee/OneDrive - SMAGEE" --remote-schema "plink.exe -i smagee-ubuntu.ppk %s rdiff-backup --server" "C:/Users/smagee" smagee@10.136.128.6::/home/smagee/Backups/smagee-pc

The result was a successful RDiff-Backup.

You can also exclude directories using the example above.

I hope this helps.  I found the following site very helpful in putting the above together.

Improving on backing up with rsync

http://linuxrhino.blogspot.co.uk/2011/04/backing-up-with-rdiff-backup.html

 

 

 

 

Create a self signed certificate and sign a PowerShell file

I believe some of the functionality from the commands below is only available in PowerShell v5 and newer.

Launch PowerShell and start by creating your code signing certificate.

New-SelfSignedCertificate -certstorelocation cert:\CurrentUser\my -dnsname "Sam Magee" -Type CodeSigningCert

This should generate a thumbprint, you’ll need this later.

ThumbprintSubject

66EWR4R0997G7UY9JRTY3J4IU5UI6LO2 CN=Sam Magee

Convert a plain text string (a secure password) into a secure string and then save the secure string into a variable.

$pwd = ConvertTo-SecureString -String "Password" -Force -AsPlainText

Export the pfx file, substitute the location, filename and thumbprint for your own

Export-PfxCertificate -Cert Cert:\CurrentUser\My\66EWR4R0997G7UY9JRTY3J4IU5UI6LO2E18F039BC -FilePath "D:\Sam’s-Cert.pfx" -Password $pwd

Export the crt file, substitute the location, filename and thumbprint for your own.  This is the file you will install on the machine you want to run your signed PowerShell code.

Export-Certificate -Cert Cert:\CurrentUser\My\66EWR4R0997G7UY9JRTY3J4IU5UI6LO2E18F039BC -FilePath "D:\Sam’s-Cert.crt"

Now that you have your PFX file exported, double click on it to reinstall it on this machine.  Install it for the Current User to overwrite the certificate we’ve just made.  When asked to fill in the “Import options” tick the following:

  • Enable strong private key protection.  You will be prompted every time the private key is used by an application if you enable this option
  • Mark the key as exportable.  This will allow you to backup or transport your keys at a later time.
  • Include all extended properties.

The top tick box will prevent malicious applications from being able to sign files as you.  The second will allow you to port your key to another machine if needed or if you wipe your machine.  The final option keeps the setting that tells the certificate to be a code signing certificate.

Get your code signing certificate from your cert volume.

$cert = @(Get-ChildItem Cert:\CurrentUser\My -codesigning)[0]

Sign the PowerShell file with the certificate.

Set-AuthenticodeSignature "D:\test.ps1" $cert

Remote management of non-domain joined servers

https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/remote_host_management

At home I have a Windows 10 workstation with hyper-v installed and a Windows 10 laptop I want to remote manage the workstations hyper-v from.

On the Hyper-V host to be managed, run the following as an administrator:

Enable-PSRemoting

Enable-PSRemoting will create the necessary firewall rules for private network zones. To allow this access on public zones you will need to enable the rules for CredSSP and WinRM.


Enable-WSManCredSSP -Role server

On the managing PC, run the following as an administrator:

Set-Item WSMan:\localhost\Client\TrustedHosts -Value "fqdn-of-hyper-v-host"

Enable-WSManCredSSP -Role client -DelegateComputer "fqdn-of-hyper-v-host"

Additionally you may need to configure the following group policy: Computer Configuration | Administrative Templates | System | Credentials Delegation | Allow delegating fresh credentials with NTLM-only server authentication

Click Enable and add wsman/fqdn-of-hyper-v-host

Windows Server Deduplication not freeing up space after files are deleted

If you have data deduplication setup on a Windows server volume and you delete a files the space that file took up is not returned as free space until something called garbage collection has run.

 

This should run once every week.  Mine runs at 02:45 on Saturday mornings, this was the default and since it doesn’t conflict with any other high disk or CPU tasks I’ve left it at that.  You can see when deduplication tasks are scheduled by running Get-DedupSchedule.

 

To force a garbage collection run the following PowerShell:

Start-DedupJob -Type GarbageCollection -Priority normal -Volume D:

You can substitute the volume D: for the volume you want to work on.  Once you’ve run the command you can run Get-DedupStatus to see how the task is progressing.  Deduplication is CPU and disk intensive.

 

TechNet: Install and Configure Data Deduplication

The Recycle Bin on D:\ is corrupted. Do you want to empty the Recycle Bin for this drive?

https://social.technet.microsoft.com/Forums/windows/en-US/f9c89952-0c61-4170-964a-6d7e90ee7347/recycle-bin-corrupted?forum=winservergen

“i faced the similar problem earlier and i ran the below command

rd /s /q C:\$Recycle.bin

This clears out the $Recycle.bin folder from the C:\ partition. You may have to do this for each hard disk partition in your system”

The results were instant, the missing 107GB showed up on the D drive after removing the directories on the C and D drives.