Tuesday, January 17, 2017

powershell: filtering for unique lines of csv

Scenario: You have a large csv file (several hundred meg or more) representing username to computer name mappings.  The data contains a lot of duplicates as it represents activity over a period of time.  The data is already sorted by time, so how do you get the most recent activity per computer while ignoring the rest?

Pipeline method with commandlets:

import-csv .\data.csv| select -Unique computername|ConvertTo-Csv -NoTypeInformation
 |out-file .\filtered-data.csv

This ran for hours, hit several hundred MB of ram usage and eventually had to be cancelled as it was taking too long.  Unfortunately for the unique filtering on select, it had to do csv conversions to get the attribute that I wanted to filter on.


Hackish method with hash table:

$ht = new-object hashtable

function selective-add {
 [CmdletBinding()]
   param ( [Parameter(Mandatory=$True,ValueFromPipeline=$True)]$line )
   begin {}
   process {
     $data = $line.split(',')
     if ($ht.contains($data[1])) {} else {
       $ht.add($data[1],$data[0])
     }
   }
}

get-content .\data.csv | selective-add
$ht.keys | % { add-content -path filtered-data.csv -value $("{0},{1}" -f $_, $ht.Item($_)) }

This only took about 15 minutes, however it sucked up twice as much ram as the previous method in a very short period of time.

Tuesday, December 27, 2016

Fixing dns record permissions for dynamic dns

In case you have dns records that need to be changed from static to dynamic, or the machine/clusters that will be updating them have changed, you can modify the dns record permissions to allow updates.  Doing this through the gui is fine for a few records, but if you need something a bit more simple and automated, you can try this script.  It takes both the computer name (doesn't have to match the dns record, just needs to be a computer object), and the dns record (fqdn) that you are updating.  This script will work for AD dns only, and would be limited to the current domain, and likely any forest wide partitions.  If you check my article on managing other domains in powershell, you can probably get edits to other domains working on this with a few modifications and extra parameters.



param (
  $computername,
  $dnsrecord
)

$script:computernameSam = $computername + "$"

try {
  import-module activedirectory
} catch {
  write-error "This script requires the AD powershell module"
  exit
}
while ( (test-path -path Ad:) -ne $true  )
{
  start-sleep -seconds 2
}

#Standard ACL for a dynamic dns entry
  #ActiveDirectoryRights : CreateChild, DeleteChild, ListChildren, ReadProperty, DeleteTree, ExtendedRight, Delete,
  #                        GenericWrite, WriteDacl, WriteOwner
  #InheritanceType       : None
  #ObjectType            : 00000000-0000-0000-0000-000000000000
  #InheritedObjectType   : 00000000-0000-0000-0000-000000000000
  #ObjectFlags           : None
  #AccessControlType     : Allow
  #IdentityReference     : Domain\machine$
  #IsInherited           : False
  #InheritanceFlags      : None
  #PropagationFlags      : None
# 

function get-partition {
  param ( $record )
  #need to split off everything after first name to find longest zone match
  #using get-dnsserverzone ($name)
  
  $dnsrecordparts = $record.split(".")
  
  for ($i = 1; $i -lt $dnsrecordparts.length; $i++) {
    $zonenameTest = $dnsrecordparts[$i..($dnsrecordparts.length -1)] -join "."
    $zoneObj = get-dnsserverzone $zonenameTest -ea 0
    if ($zoneObj -ne $null) {
    write-output -inputobject $zoneobj
    $i = $dnsrecordparts.length + 1
    }
  }
  
}

function get-dnsobject {
  param ($record)
  $zoneObject = get-partition -record $record
  if ($zoneObject -ne $null) {
    $zonename = $zoneObject.zonename
    $record -match "(.*)(\.$zonename)"
    $dnsRecordDN = "dc=" + $matches[1] + "," + $zoneobject.distinguishedname
     try {
       get-adobject  $dnsRecordDN
    } catch { Throw "Unable to find dns record for this machine"}
  } else { throw "DNS zone not found"}
}

try {
  try {
    $guid = [guid]'00000000-0000-0000-0000-000000000000'
    $adcomputer = get-adcomputer $computername -property objectsid
    $sid = $adcomputer.objectsid
    $ctrl = [System.Security.AccessControl.AccessControlType]::Allow
    $rights = 983423
    $intype =[System.DirectoryServices.ActiveDirectorySecurityInheritance]::None
    $rule = New-Object System.DirectoryServices.ActiveDirectoryAccessRule($sid,$rights,$ctrl,$guid)
  } catch { throw "Unable to get computer account SID" }

  try {
    #find record
    $dnsDN = get-dnsObject -record $dnsrecord
  } catch { throw $_ }
  
  try {
    $acl = get-acl ad:"$($dnsDN.distinguishedname)"
    $acl.setowner([system.security.principal.ntaccount]"$script:computernameSam")
    $acl.AddAccessRule($rule)
    Set-Acl -acl $acl -path ad:"$($dnsDN.distinguishedname)"
  } catch { throw $_ }

} catch { $_}

Wednesday, December 21, 2016

DNS - limited forwarding delegated subdomain (in AD integrated zone)

If you find yourself in need to creating subdomains off of an existing AD integrated zone, which forward externally, you may encounter the problem that not all AD dns servers can access an external set of servers. To limit the domain controllers that do dns forwarding, while still allowing all dc's to know how to resolve records, you can combine delegated subdomains and conditional forwarders (non-AD integrated).

Example:

-Organization has AD integrated zone Contoso.com on all AD domain controllers.
-Organization is outsourcing dns for subdomain hosting.contoso.com to external domain name servers
-Only 2 out of 50 domain controllers can access external dns for name resolution.  All others do general forwarding to these 2 domain controllers.

Problem:
1) If we create a delegation in contoso.com directly to the external dns servers, recursion is not available and name resolution is not going to happen.
2) if we create AD integrated conditional forwarding for the subdomain, all servers will try to forward to external dns and will be unable to do so, causing queries to fail

Solution:
-Create a subdomain delegation in contoso.com using only the name servers of the 2 internal domain controllers that have access to forward dns queries to external servers
-Create non-AD integrated conditional forwarders for hosting.contoso.com on these same 2 servers, which use the dns server IP's of the external dns provider
-On the external provider, set up a dns zone for hosting.contoso.com

Wednesday, December 14, 2016

Learn typing on arabic keyboards

When trying to find a decent place to learn how to type arabic, I found a lot of sad looking websites that either didn't have what the search results suggested, or they only showed a few phrases to practice on. Most are more oriented as online keyboards for you to type and paste with. Since most operating systems provide alternate keyboards and easy switching between them, for anyone who seriously needs to type arabic, it would be great to have some alternative. So I thought I would put together something simple with lessons that goes through a touch typing style learning methodology. You can try it here: Arabic typing - free online course.

Tuesday, December 6, 2016

SCCM windows update deployment failure 0x80240438

Windows update problem caused by web troubleshooting tools and system proxy. Netsh winhttp settings were creating a local proxy that was not causing any issues in the user context, but in the system context, it was breaking outbound web connections. SCCM logs only showed the error code, but not much useful information beyond that. WindowsUpdate.log contained a clear message on the attempted use (and failure) of a proxy.

wuauhandler.log

<![LOG[OnSearchComplete - Failed to end search job. Error = 0x80240438.]LOG]!><time=".." date=".." component="WUAHandler" context="" type="3" thread="2744" file="cwuahandler.cpp:3223">
<![LOG[Scan failed with error = 0x80240438.]LOG]!><time=".." date=".." component="WUAHandler" context="" type="3" thread="2744" file="cwuahandler.cpp:3679">

updateshandler.log

<![LOG[Updates scan completion received, result = 0x80240438.]LOG]!><time=".." date=".." component="UpdatesHandler" context="" type="1" thread="9884" file="capplicabilityhandler.cpp:100">


updatesdeployment.log

<![LOG[Job error (0x80240438) received for assignment ({guid}) action]LOG]!><time=".." date=".." component="UpdatesDeploymentAgent" context="" type="3" thread="7056" file="updatesassignment.cpp:2235">


scanagent.log

<![LOG[ScanJob({guid}): CScanJobManager::OnScanComplete- failed at CScanJob::OnScanComplete with error=0x80240438]LOG]!><time=".." date=".." component="ScanAgent" context="" type="3" thread="11024" file="utils.cpp:537">


ciagent.log

<![LOG[Failed result received from applicability handler, error = 0x80240438]LOG]!><time=".." date=".." component="CIAgent" context="" type="3" thread="7056" file="capplicabilitybroker.cpp:79">


WindowsUpdate.log

yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: Web service call failed with hr = 80240438.
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: Current service auth scheme='None'.
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: Proxy List used: '127.0.0.1:8888', Bypass List used: '(null)', Last Proxy used: '127.0.0.1:8888', Last auth Schemes used: 'None'.
yyyy-mm-dd hh:mm:ss 1068 30a4 WS FATAL: OnCallFailure failed with hr=0X80240438
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: NWS retry 1 for transient error 0x80240438
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: Nws Failure: errorCode=0x803d0010
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: There was an error communicating with the endpoint at 'http://deploymentserver:8530/ClientWebService/client.asmx'.
yyyy-mm-dd hh:mm:ss 1068 30a4 WS WARNING: The given proxy cannot be reached.

------------------------------------------
netsh winhttp>show proxy

Current WinHTTP proxy settings:
Proxy Server(s) : 127.0.0.1:8888
Bypass List : (none)

Wednesday, September 28, 2016

Masjid Nabawi (Madinah) - Tips



-Getting in the front saf.  The masjid’s actual front saf is in the older (more narrow) front section of the masjid.  Due to the rawdah and other attractions in that part of the masjid, its often very full and sectioned off.  Trying to pray inside the masjid in the front saf is nearly impossible, and if you do manage to get it, it is tightly packed and hard to focus.  Try to try get there for fajr, look at going by midnight and don’t leave for any reason (don’t break wudu).  The easier way to pray in the front is outside.  In the picture above you can see the mark on the right side of the masjid where I say its an easy front row position.  This side of the masjid is longer than the left side.  Usually there are carpets set up there, and if not, there are block lines marking rows.  There are some information screens that point out not try pray in front of a certain point to avoid being in front of the imam.  That is the line of umbrellas with the double black line running just behind them.  It is a bit warm out there, but they do have the umbrellas, and some water fans (that rarely seem to blow in your direction).  During zuhr and asr, either the umbrellas don’t filter UV or the reflection of the sun on the marble carries it, so if you’re out there for a while and get sunburn easily, make sure you put on sunscreen.  The good news is, you can still get in the front row 30 min before solat.

-Following the funeral to baqi (men only).  Since almost every solat in this masjid is followed by the funeral prayer, you can increase your rewards by following the procession into baqi and helping carry and bury the body(s).  You need to be fast for this, and in the right areas of the masjid.  The bodies come out right after solat from a door in front of where the imam is positioned.  The best place to be is on the left side of the masjid (outside), or inside and very close to Bab Bilal.  Getting to Bab Bilal from the inside of the masjid can be blocked off at times.  When they have the rawdah open to women, there will be barriers blocking you from walking in that direction.  Those will be removed at least an hour before an fard solat.  You can be on the right side (outside), but you need to run as soon as the funeral prayer is over.  If you are a bit far on the right, you may want to use the short break between the fard solat and the janazah prayer to move farther to the left and jump in a saf somewhere closer to the funeral exit.  If you’re too far behind the procession, the gates of baqi will be closed on you and you can’t get in.  The whole burial/walking takes at least 30 minutes.  It will be very dusty and dirty.  Due to the wind and the soft sand, you will end up in a mini sandstorm when the grave is being filled.  Make sure you have a water bottle with you because once you get inside baqi there’s usually a steady and very dry hot wind.

-Quran teaching.  There will be many teachers that set up quran circles in the masjid.  If I remember correctly, this started at least an hour before asr, and more towards the front right of the newer section of the masjid.  They might be in other places as well.  These are free and open classes where the teacher will invite anyone nearby to come and practice recitation.

-Halaqahs.  There is one American scholar (Tahir Wyatt) who gives a halaqah in english between magrib and isha for brothers.  This is located near the right rear brother’s entrance.  Once you go in the door, turn left and go a short distance.  The chair is at the very back of the masjid.

-Friday sermon in other languages.  If you have a radio with you (or phone that does radio), there are several translated versions of the khutbah being broadcast.  This is both in madina and mecca.  I couldn’t find a list of frequencies this time, but I believe english in madina was 105.5, and for mecca it was something around 102.  You won’t be able to find the stations very easily in advance of the beginning of the sermon, but look for radio frequencies that are broadcasting silence instead of static.  There should be at least 4 of them.

-zam zam.  I’m not 100% sure on this, but there are a lot of wudu and drinking water places around the outside of the masjid at the mini buildings that you see all around it.  I thought this might be zam zam, but most likely it is just drinking water.  There is occasionally a mobile metal water tank set up outside with taps on it which would be zam zam.  Otherwise, all throughout the masjid, they have the coolers set up.

-rawdah.  Trying to get in here before a fard solat is very difficult as people are going in there to stay.  The crowds are always large.  Trying to get in around the middle of the night for a short amount of time might be possible.  In these times, people will usually give you some space if you indicate you just want to pray a few rakats there.


-Praying inside.  The masjid fills up quickly in the back, but not so much in the front or middle.  If you don’t want to have to move forward 10 or more saf’s just as the prayer begins, pray directly behind a column.  This has the added benefit of being near aircon, and less of a chance that people will walk in front of you during any sunnah prayers you may want to pray.  You should start your position more towards the front though to help avoid being part of the “gap” problem.