Intersting kerberos and ssh troubleshot

There was an interesting kerberos troubleshoot today, someone set up a round-robin DNS solution where host lets call it server-x had a bunch of A records pointing to different IP addresses.

For illustration purposes lets say it looks like this
server-x.domain.com 15 IN A 192.168.1.1
server-x.domain.com 15 IN A 192.168.1.2
server-x.domain.com 15 IN A 192.168.1.3 
server-x.domain.com 15 IN A 192.168.1.4

server1.domain.com 300 IN A  192.168.1.1
server2.domain.com 300 IN A  192.168.1.2
server3.domain.com 300 IN A  192.168.1.3 
server4.domain.com 300 IN A  192.168.1.4

1.1.168.192 300 IN PTR server1.domain.com 
2.1.168.192 300 IN PTR server2.domain.com
3.1.168.192 300 IN PTR server3.domain.com
4.1.168.192 300 IN PTR server4.domain.com

When attempting to ssh to server-x it would sometimes work but sometimes return an error that it filed to initialize gss context.  We finally dug in and found the following

With a completely clean cache (i.e. TGT only) when failure occurred we could tell the server1 was being contacted to but the cache contained a service ticket for server 2.  It turned out that ssh would do it's own resolution separate from GSSAPI's canonicalisation.  The work around we found was to wrap the call in some script that first resolves the name and passes it to ssh.  This way both ssh and GSSAPI skip the resolution step.

A bit later my colleague discovered an option in ssh called GSSAPITrustDns, which makes sure that the name is resolved only once by ssh and then is passed to gssapi, preventing the double resolution.

The longer answer is that if you must use kerberos behind a load balancer do not use round-robin, in fact round-robin is a pretty bad load balancer for just about anything, kerberized or not

A little on verilog

So I am trying to learn verilog, I have a conter,
reg counter;
And am expecting it to reach 15 at some point, but it never does it goes between 1 and 0.  It works when I declare counter as a integer everything works.  Well reg by default is just 1 bit adding 1+0 = 1 and 1+1=0, declaring as reg [31:0] counter fixed the problem.

error:0B084009:x509 certificate routines:X509_load_cert_crl_file:PEM lib

Started getting this error today after renewing my fake CA cert im using to test out an app.  No it wasn't the new lines, the problem was a missing '-' character at the end of the last line.

This is what happens when you just copy and paste the cert

Borland C++

Moving can be fun you find all these forgotten gems.  One last look before I throw these out

Russian cryptographic primitives -- the actors

Чебурашка <- Alice
Гена <- Bob
Шапокляк <- Eve

CRC32

Another piece fell into place

calc_adler32 :: String->Int->Int->Int

calc_adler32 [] a b = (b `shiftL` 16) .|. a

calc_adler32 str a b  = 

  let ap = (a + (ord $ head str)) `mod` mod_adler

      bp = (ap + b) `mod` mod_adler

  in calc_adler32 (tail str) ap bp

    

crc32 :: String -> Int

crc32 str = calc_adler32 str 1 0

*Main> crc32 "abc"
38600999

Making shingles

This is starting to get a little boring, it seems that all of my Haskell work consists of recombining lists and folding them, but either way yet another piece of the puzzle is now in place, I can now generate shingles


makeShingle :: [[a]]->[[a]]
makeShingle [] = []
makeShingle zs 
  | length zs >= 4 = (foldl(\acc z -> (acc ++ z)) [] (take 4 zs)) : (makeShingle $ drop 1 zs)
  | otherwise = []



makeShingle $ filter(/="") $ genericRev $ otherTok "this, is a string.  it is a string, like many other.  but i like it, i like it a lot!" standard_delim
["thisisastring","isastringit","astringitis","stringitisa","itisastring","isastringlike","astringlikemany","stringlikemanyother","likemanyotherbut","manyotherbuti","otherbutilike","butilikeit","ilikeiti","likeitilike","itilikeit","ilikeita","likeitalot"]

Set permutations

There is a part of my project that calls for a permuted set, that is given the original set of elements create a set of sets where each set contains a permutation of the original.  It turns out that this is more or less simple once you break the problem down into smaller parts.

There are a few algorithms to generate permutations,  I chose interleaving for it being relatively simple and straight forward.
For a set of [x,y, x] permute the first two elements
[[x, y], [y, x]]
and then interleave each set with z
[[z, x, y], [x, z, y], [x, y, z], [z, y, x], [y, z, x], [y, x, z]]

First we create a function that would take a list an a value and create permutations of that list with a value:

interleaveSet :: [a] -> a -> [[a]]
interleaveSet [] value = [[value]]
interleaveSet xs value = [ take pos xs ++ [value] ++ drop pos xs | pos <- [0..length(xs)] ]

Here are some samples:

*Main> interleaveSet [] 1
[[1]]
*Main> interleaveSet [1] 2
[[2,1],[1,2]]
*Main> interleaveSet [1,2,3,4] 5
[[5,1,2,3,4],[1,5,2,3,4],[1,2,5,3,4],[1,2,3,5,4],[1,2,3,4,5]]

So we now have a way of interleaving a set with a value, but remember that what we need to do is build this from an existing set.  What we need is something that will take the output of interleaveSet and interleave the sets created by it with the next value.  I was at first tempted to use a list comprehension once again

interleaveSets sets value = [ interleaveSet set value | set <- sets ]

but since interleaveSet outputs a [[a]] and interleaveSets will create a list of those, our return value will be [[[a]]], which is not what we want at all.


*Main> interleaveSets (interleaveSet [1,2,3,4] 5) 6
[[[6,5,1,2,3,4],[5,6,1,2,3,4],[5,1,6,2,3,4],[5,1,2,6,3,4],[5,1,2,3,6,4],[5,1,2,3,4,6]],[[6,1,5,2,3,4],[1,6,5,2,3,4],[1,5,6,2,3,4],[1,5,2,6,3,4],[1,5,2,3,6,4],[1,5,2,3,4,6]],[[6,1,2,5,3,4],[1,6,2,5,3,4],[1,2,6,5,3,4],[1,2,5,6,3,4],[1,2,5,3,6,4],[1,2,5,3,4,6]],[[6,1,2,3,5,4],[1,6,2,3,5,4],[1,2,6,3,5,4],[1,2,3,6,5,4],[1,2,3,5,6,4],[1,2,3,5,4,6]],[[6,1,2,3,4,5],[1,6,2,3,4,5],[1,2,6,3,4,5],[1,2,3,6,4,5],[1,2,3,4,6,5],[1,2,3,4,5,6]]]

Since we want to be able to feed it back to interleaveSets, this is not what we want.

This will do:

interleaveSets :: [[a]] -> a -> [[a]]
interleaveSets [] value = [[]]
interleaveSets (s:sets) value = interleaveSet s value ++ interleaveSets sets value

*Main> interleaveSets (interleaveSet [1,2,3,4] 5) 6
[[6,5,1,2,3,4],[5,6,1,2,3,4],[5,1,6,2,3,4],[5,1,2,6,3,4],[5,1,2,3,6,4],[5,1,2,3,4,6],[6,1,5,2,3,4],[1,6,5,2,3,4],[1,5,6,2,3,4],[1,5,2,6,3,4],[1,5,2,3,6,4],[1,5,2,3,4,6],[6,1,2,5,3,4],[1,6,2,5,3,4],[1,2,6,5,3,4],[1,2,5,6,3,4],[1,2,5,3,6,4],[1,2,5,3,4,6],[6,1,2,3,5,4],[1,6,2,3,5,4],[1,2,6,3,5,4],[1,2,3,6,5,4],[1,2,3,5,6,4],[1,2,3,5,4,6],[6,1,2,3,4,5],[1,6,2,3,4,5],[1,2,6,3,4,5],[1,2,3,6,4,5],[1,2,3,4,6,5],[1,2,3,4,5,6],[]]

I am not sure what that empty element at the end is, will have to deal with it later (I am sure it is a side effect of my base case)

Ah the solution is that the base case return a [] because any set ++ [] is just set

*Main> [1,2,3] ++ []
[1,2,3]

And here is the new result:
*Main>  interleaveSets (interleaveSet [1,2,3,4] 5) 6
[[6,5,1,2,3,4],[5,6,1,2,3,4],[5,1,6,2,3,4],[5,1,2,6,3,4],[5,1,2,3,6,4],[5,1,2,3,4,6],[6,1,5,2,3,4],[1,6,5,2,3,4],[1,5,6,2,3,4],[1,5,2,6,3,4],[1,5,2,3,6,4],[1,5,2,3,4,6],[6,1,2,5,3,4],[1,6,2,5,3,4],[1,2,6,5,3,4],[1,2,5,6,3,4],[1,2,5,3,6,4],[1,2,5,3,4,6],[6,1,2,3,5,4],[1,6,2,3,5,4],[1,2,6,3,5,4],[1,2,3,6,5,4],[1,2,3,5,6,4],[1,2,3,5,4,6],[6,1,2,3,4,5],[1,6,2,3,4,5],[1,2,6,3,4,5],[1,2,3,6,4,5],[1,2,3,4,6,5],[1,2,3,4,5,6]]

And finally to put it all together:

permuteSet :: [a] -> [[a]]
permuteSet [] = []
permuteSet set = foldl(\acc e -> interleaveSets acc e) seed (tail set)
  where seed = interleaveSet [] (head set)

Lets test

*Main > permuteSet [1,2,3,4]
[[4,3,2,1],[3,4,2,1],[3,2,4,1],[3,2,1,4],[4,2,3,1],[2,4,3,1],[2,3,4,1],[2,3,1,4],[4,2,1,3],[2,4,1,3],[2,1,4,3],[2,1,3,4],[4,3,1,2],[3,4,1,2],[3,1,4,2],[3,1,2,4],[4,1,3,2],[1,4,3,2],[1,3,4,2],[1,3,2,4],[4,1,2,3],[1,4,2,3],[1,2,4,3],[1,2,3,4]]

Looks about right

*Main> length $ permuteSet [1,2,3,4]

24

Number of permutations is equal to n! / (n -k)!

Where k is the number of elements from n per permutation, in our case k = n so that turns into 0! which we know is one

and the factorial of 4 is 4 * 3 * 2 *1 = 24

BASH directory recursion, scope and local variables

How quaint, I am using a bash shell script to do backups of my data, very similar to time machine if anyone is familiar.  Not sure why shell, at this point I can count on so many other things to be included, but I chose shell.  My directory recursion was failing after a few loops and I could not figure out why, until I found out that bash does not do lexical scope very well, and local variables need to be declared as local explicitly.  This way when a recursive invocation exits it's assignments will not live on.

A more readable haskell string tokenizer

Another crack and a tokenizer

otherTok :: String -> [Char] -> [[Char]]
otherTok [] _ = []
otherTok cs delim = foldl(\acc c -> if c `elem` delim then [] : acc else (head acc ++ [c]) : (tail acc) ) [] cs

*Main> otherTok "blah,blah blah. blah! blah" " ,.!"
["blah","","blah","","blah","blah","*** Exception: Prelude.head: empty list

Encountering a delimiter char produces an empty string, which I can remove later with filter, not sure what to do with the Exception.

P.S. Problem solved

otherTok :: String -> [Char] -> [[Char]]
otherTok [] _ = []
otherTok cs delim = foldl(\acc c -> if c `elem` delim then [] : acc else (head acc ++ [c]) : (tail acc) ) [""] cs

*Main> filter (/="") (otherTok "blah,blah blah. blah! blah" " ,.!")
["blah","blah","blah","blah","blah"] 

P.S.S The tokens that result are in reverse order of how the words show up in a line, also easily fixed 

rev :: [a] -> [a]
rev [] = []
rev xs = foldr(\x acc -> acc ++ [x]) [] xs 
 *Main> rev(filter(/="") (otherTok "abc,def.ghi" ",."))
["abc","def","ghi"] 

Or just use the builotin reverse and stop trying to reinvent the wheel

Mac OS finder/account slowness

I've noticed that it takes finder a good 15 seconds to startup sometimes.  None of the forums I visited had any good answers, most dealt with generic account slowness and most recommended to just create a new account.  I decided to run top and see what is running and process flags when I noticed at the top of the list mount and mount_nfs sleeping.

PID   COMMAND      %CPU TIME     #TH  #WQ  #POR #MRE RPRVT  RSHRD  RSIZE  ...

2429  mount_nfs    0.0  00:00.00 1    0    15   28   112K   288K   428K   17M    586M   2385 2428 sleeping ...

2428  mount        0.0  00:00.00 1    0    14   26   104K   284K   404K   9496K  578M   2385 2385 sleeping 

(output  discarded for brevity)

180   Finder       0.0  01:35.63 4    1    213  522  18M    51M    62M    37M    908M   180  174  sleeping 

I booted up my NFS server (which for my own reasons I do not keep constantly powered on) and tried launching finder again and this time, the window popped up with no delay. 

Shut off the NFS server, closed and reopened finder just to make sure that this delay is repeatable and reproducible.  Indeed it is and every time I open a finder window Mac OS (not sure which one of the internals) will attempt to mount_nfs.

Added the following to my NFS mount options
-dumbtimer -timeo=3

Closed the finder window clicked on the finder icon again, this time the delay was much shorter.  Hope this helps someone.

P.S. Having solved this problem does not make me a mac convert.  Figuring out why it takes finder so long to list a directory when ls only takes a split second might get me half way there.


P.S.S. Also deleting related items from /Users/$USER/Library/Preferences


Finder:
com.apple.finder.plist

Digital Photo Professional:

com.canon.Digital Photo Professional.LSSharedFileList.plist

com.canon.Digital Photo Professional.plist

helped, and now they work well even without NFS.  So the forums were right about that, just pointed to a different place when they mentioned preference cache.  So in my case I found the root cause, but resolved it the same way everyone else did.

HP-UX getent

I spent quite a bit of time looking for a getent equivalent for HP-UX.  I needed something that would return a non zero value if a user or a group I queried were not there.  In case you are having as much fun as I am, the commands are pwget and grget

haskell string tokenizer

I recently needed a function to split strings on a delimeter and found this one on http://blog.julipedia.org/2006/08/split-function-in-haskell.html:


tokenizeString :: String -> [Char] -> [[Char]]
tokenizeString [] _ = [""]
tokenizeString (c:cs) delim
  | c `elem` delim = [] : rest
  | otherwise = (c : head rest) : tail rest
    where rest = tokenizeString cs delim

It took me a little while to wrap my mind around the recursion and what goes on here, a pen and some paper and trace it out this is really really cool.   It's a shame that I was not able to come up with this on my  own.

Execution trace original string: "a,b" delimiter list: ","

I shortened some names to save space by hMt is really head $ mT and tmT is tail $ mT where mT is myTokenizer.  I am no expert, but the secret sauce appears to be in the base case which always returns a [""] -- list of strings.  And how the string elements are concatenated:

Char : [Char]: [[Char]]  -- Single character to a string to a list of strings

Fun with Haskell

Had some more time to play with Haskell today, I am still very new at this, not being able to use iteration makes coding "interesting".  Also this makes nesting data structures a little challenging.

For example take a function that returns a list:
multAddFunc :: Integer -> Integer -> [Integer]
multAddFunc a b = [a*b, a+b]

*Main> multAddFunc 2 3
[6,5]

Now lets say we had two lists of numbers [1,2,3] [4,5,6] And we wanted to write a function that processed them:
*Main> [ multAddFunc x y | x<-[1,2,3], y<-[4,5,6] ]
[[4,5],[5,6],[6,7],[8,6],[10,7],[12,8],[12,7],[15,8],[18,9]]

This just returned a list of lists, observe

*Main> :t [ multAddFunc x y | x<-[1,2,3], y<-[4,5,6] ]
[[4,5],[5,6],[6,7],[8,6],[10,7],[12,8],[12,7],[15,8],[18,9]]

But what if we wanted a flattened list?  For this we would have to concatenate the lists together using the ++ operator

*Main> :t (++)

(++) :: [a] -> [a] -> [a]

listMath :: [Integer] -> [Integer] -> [Integer]

listMath [] [] = []

listMath (x:xs) (y:ys) = (multAddFunc x y) ++ (listMath xs ys)

*Main> listMath [1,2,3] [4,5,6]

[4,5,10,7,18,9]

And there we have it a flattened list

Shingles

It is easy to create a naive Data Leakage Protection (DLP) Product that will look for exact data or pattern matches, it is a lot more difficult to spot similarity between documents such as this document is x% similar to this reference.  This article looks interesting and the approach seems easy to implement  http://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html