Thursday, December 26, 2013

Extra data section and different comparison algorithm

Extra data section:

A new section was added to the right of the heatmap. When a rectangle from the heatmap is selected, a new section appears on the right that displays additional information regarding the two scripts compared. It displays the alphabets of the scripts, their similarity score on a scale normalized to a range from 0 to 1 (see code below), and a world map that will display the areas in the world where the scripts are used.


New formula:

I changed the formula used to create the similarity score to an Euclidian distance formula instead of the difference accumulation. Using this formula is comparable in results to the previous method (see validation results and new heatmap below) with has slightly better results. It makes char clustering much simpler, more on clustering in the next blog post.
The new formula highest difference (Greek and Telugu as before) is ~21.5. Normalized to 1, where 1 is identity, the biggest difference received in this validation (Tamil as in previous validation) is ~0.83 with the new formula (previous ~0.81) and best score (Greek) is ~0.94 (previously Hebrew with ~0.93).

Telugu :  1.72155494908
Cyrillic :  1.31314939211
Greek :  1.17887522459
Malayalam :  2.3690800984
Thai :  1.69981410346
Latin :  1.28629859902
Gujarati :  2.14389522695
Hebrew :  1.2465182772
Devanagari :  1.60605141708
Arabic :  1.7936517922
Tamil :  3.58195406289

Images and code:

Visualization with extra data section and new weight formula:




Normalize similarity scores:

# Normalize weights to be from 0 (identity) to 1 (greatest difference)
for row in data_arr:
    for entry in row:
        value = entry[0]
        entry[0] = 0 if value == maxData else (value - maxData)*(-1)/maxData

Euclidian distance:

def getDiffFromDictionary(char1, char2):
    char1_lines = float(char1['evarage_lines'])
    char1_curves = float(char1['evarage_curves'])
    char2_lines = float(char2['evarage_lines'])
    char2_curves = float(char2['evarage_curves'])
    return distEclud(char1_lines, char1_curves, char2_lines, char2_curves)

def distEclud(x1, y1, x2, y2):
    return sqrt(power(x1 - x2, 2) + power(y1 - y2, 2))

No comments:

Post a Comment