Converting model from Caffe to ONNX - artificial-intelligence

I'd like to prototype with Javascript to detect the sky using a model trained on a SkyFinder dataset.
I tried to convert the Caffe model (prototxt and trained data above) published here to the ONNX model using MMdnn.
Following command executed,
mmconvert --srcFramework caffe --inputWeight baseline.caffemodel --inputNetwork deploy.net --dstFramework onnx --outputModel baseline.onnx --inputShape 1 3 240 320
and this error came out
mmdnn.conversion.caffe.errors.ConversionError: Layer not found: label_0
Below is an excerpt from the first part of prototxt above. label_0 is connected to the dummy_data and convine_1 layers but does not appear to be defined anywhere.
name: "DeepBlueSky"
input: "data"
input_dim: 1
input_dim: 3
input_dim: 240
input_dim: 320
layers {
name: "dummy_data"
type: DUMMY_DATA
top: "label_0"
dummy_data_param {
data_filler {
type: "constant"
std: .5
}
num: 1
channels: 1
height: 240
width: 320
}
}
layers {
name: "combine_1"
type: CONCAT
bottom: "data"
bottom: "label_0"
top: "combine_1"
concat_param {
concat_dim: 1
}
}
...
I'm a newbie, so it's possible I'm missing the basics, but if you've had a similar experience and have been able to resolve it, would you please share it?

Related

FBX - In a FBX file, which data should I use to build the skeleton of a model and animate it?

For learning purposes, I'm writing a FBX file reader in c++, in order to understand how this kind of 3D models are working. I must specify that I already successfully wrote a .x file reader in the past, so I'm comfortable with concepts like skeleton, bones and skin weights.
At this point I wrote a code able to read and parse the FBX file content, and extract and show the basic model in its T pose. I also assume that I correctly understood the way the FBX data are structured, and how to link the elements together and rebuild the important hierarchies like the skeleton.
However I'm a little puzzled about which data I need to use for the bones and animations. About the first ones I have a hierarchy of deformers, which each contains more or less data like that:
Deformer: 46748048, "SubDeformer::clavicle_r_adult_female", "Cluster" {
Version: 100
UserData: "", ""
Indexes: *519 {
a: 1988,2000,2001,2002,2003,7898,7899,7900,7901,7902,7903,7904,7905,7906,7907,7908,7909,7910,7911,7912,7913,7914,7915,7916,7917,7918,7919,7920,7921,7922,7923,7924,7925,7926,7927,7928,7929,7930,7931,7932,7933,7934,7935,7936,7937,7938,7939,7940,7941,7942,7943,7944,7945,7946,7947,7948,7949,7950,7951,7952,7953,7954,7955,7956,7957,7958,7959,7960,7961,7962,7963,7964,7965,7966,7967,7968,7969,7970,7971,7972,7973,7974,7975,7976,7977,7978,7979,7980,7981,7982,7983,7984,7985,7986,7987,7988,7989,7990,7991,7992,7993,8000,8001,8002,8003,8004,8005,8012,8013,8014,8015,8016,8017,8024,8025,8026,8027,8028,8029,8030,8031,8032,8033,8034,8035,8036,8037,8038,8039,8040,8041,8042,8043,8044,8045,8046,8047,8048,8049,8050,8051,8052,8053,8054,8055,8056,8057,8058,8059,8060,8061,8062,8063,8064,8065,8066,8067,8068,8069,8070,8071,8072,8073,8074,8075,8076,8077,8078,8079,8080,8083,8084,8085,8086,8087,8088,8089,8090,8093,8094,8095,8096,8097,8098,8099,8100,8101,8102,8103,8104,8105,8106,8107,8108,8109,8110,8111,8112,8113,8114,8115,8116,8119,8120,8121,8122,8123,8124,8125,8126,8127,8128,8129,8130,8131,8132,8133,8134,8135,8136,8137,8138,8139,8140,8141,8142,8143,8144,8145,8146,8147,8148,8149,8150,8151,8152,8153,8154,8155,8156,8157,8158,8159,8160,8161,8162,8163,8166,8167,8168,8169,8170,8171,8172,8173,8174,8175,8176,8177,8178,8179,8180,8181,8182,8183,8184,8185,8186,8187,8188,8189,8190,8191,8192,8193,8194,8195,8196,8197,8198,8199,8200,8201,8202,8203,8204,8205,8206,8207,8208,8209,8210,8211,8212,8213,8214,8215,8216,8217,8218,8219,8220,8221,8222,8223,8224,8225,8226,8227,8228,8229,8230,8231,8232,8233,8234,8235,8236,8237,8238,8240,8241,8242,8246,8247,8248,8252,8253,8254,8258,8259,8260,8264,8265,8266,8270,8271,8272,8273,8274,8275,8276,8277,8278,8279,8280,8281,8282,8283,8284,8285,8286,8287,8288,8289,8290,8291,8294,8295,8296,8297,8298,8299,8300,8301,8302,8304,8305,8306,8307,8308,8309,8310,8311,8312,8313,8314,8315,8316,8317,8318,8319,8320,8321,8322,8323,8324,8325,8326,8327,8328,8329,8330,8331,8332,8333,8334,8335,8336,8337,8338,8339,8340,8341,8342,8343,8344,
8345,8346,8347,8348,8349,8350,8352,8353,8354,8355,8356,8357,8358,8360,8361,8362,8431,8432,8433,8434,8463,8464,8465,8466,8467,8478,8479,8480,8481,8482,8483,8494,8495,8496,8497,8498,8499,8507,8508,8509,8510,8511,8512,8513,8514,8515,8523,8524,8525,8526,8527,8528,8529,8530,8531,8539,8540,8541,8542,8543,8547,8549,8552,8553,8556,8559,8560,8563,8564,10356,10357,10358,10359,10360,10361,10362,10363,10364,10365,10366,10367,10368,10369,10370,10371,10372,10373,10374,10386,10387,10388,10393,10394,10395,10396,10397,10398,10399,10449,10450,10451,10456,10457,10458,10459,10460,10461,10483,10548,10842,10847,10852
}
Weights: *519 {
a: 0.00240023992955685,0.00430000014603138,0.00350034981966019,0.00120000005699694,0.000200019989279099,0.00980097986757755,0.00499999988824129,0.01970000192523,0.00990098994225264,0.035707138478756,0.0185018517076969,0.0623937658965588,0.033396665006876,0.105110518634319,0.0757924243807793,0.179082110524178,0.13910000026226,0.013798619620502,0.013399999588728,0.0275055021047592,0.0269000008702278,0.0493950620293617,0.048200000077486,0.0841084122657776,0.0820082053542137,0.136999994516373,0.13340000808239,0.226000010967255,0.220100000500679,0.0472999997437,0.0808000043034554,0.133586660027504,0.224577531218529,0.0017001701053232,0.00480048032477498,0.000899909995496273,0.0110988905653358,0.00350035005249083,0.00960096064954996,0.00170000013895333,0.0222022216767073,0.00680136028677225,0.0290029011666775,0.0170999988913536,0.104789525270462,0.00340068014338613,0.0397000014781952,0.0133013306185603,0.0502999983727932,0.0290999989956617,0.0659065991640091,0.00659999996423721,0.0672932714223862,0.0258000008761883,0.0838000029325485,0.0470000021159649,0.0417041704058647,0.0129000004380941,0.10980000346899,0.0503949634730816,0.142028406262398,0.0764000043272972,0.0248000007122755,0.0252025201916695,0.182218223810196,0.00970000028610229,0.0192980710417032,0.0339000001549721,0.0555944368243217,0.0873000025749207,0.139286071062088,0.00959999952465296,0.0192980710417032,0.033500000834465,0.0545000024139881,0.0843999981880188,0.131599992513657,0.00850085075944662,0.0170999988913536,0.029399998486042,0.0468953102827072,0.0710999965667725,0.108689121901989,0.00689999992027879,0.0139013901352882,0.0242999996989965,0.0398000031709671,0.0597000010311604,0.0888999998569489,0.00240000011399388,0.00469999993219972,0.00889822095632553,0.015498448163271,0.0262026209384203,0.0452045202255249,0.000400000018998981,0.000799920002464205,0.00160016003064811,0.0031999999191612,0.00650000013411045,0.0130000002682209,0.00789999961853027,0.00700070010498166,0.0487999990582466,0.0830999985337257,0.13629999756813,0.227800011634827,
0.0131986793130636,0.0136986300349236,0.0263999961316586,0.02730000205338,0.0160016007721424,0.0139013901352882,0.100089997053146,0.0858914107084274,0.143514364957809,0.236076399683952,0.288500010967255,0.223422348499298,0.121500000357628,0.0781000033020973,0.0348000042140484,0.0204979497939348,0.260600000619888,0.183699995279312,0.206399992108345,0.403200000524521,0.463299989700317,0.348899990320206,0.164700001478195,0.108710870146751,0.0527999997138977,0.0400959886610508,0.430843085050583,0.40464049577713,0.339600026607513,0.470752894878387,0.442299991846085,0.31470000743866,0.184518456459045,0.134086593985558,0.0837000012397766,0.0749074891209602,0.456645667552948,0.442955672740936,0.403840392827988,0.44200000166893,0.377400010824203,0.323167681694031,0.225322529673576,0.175200000405312,0.125200003385544,0.104189582169056,0.47350001335144,0.480800032615662,0.467946827411652,0.413058698177338,0.0705070570111275,0.062993697822094,0.0500949919223785,0.0372037179768085,0.366763323545456,0.0221999995410442,0.0318000018596649,0.041700005531311,0.0393960624933243,0.0296000000089407,0.011001099832356,0.00550054991617799,0.00270054000429809,0.00970000121742487,0.0124000003561378,0.0203000027686357,0.0268973093479872,0.0223999992012978,0.00740000000223517,0.00340000004507601,0.00170016998890787,0.00490000005811453,0.00620000017806888,0.0101010100916028,0.0134013397619128,0.0115000000223517,0.00360036012716591,0.0459000021219254,0.0630936920642853,0.0871087089180946,0.120312035083771,0.1317999958992,0.103510357439518,0.072500005364418,0.0460953935980797,0.0194980520755053,0.0107010696083307,0.0157999992370605,0.0253025311976671,0.0337000004947186,0.0337999984622002,0.0220999997109175,0.0142985703423619,0.00709929037839174,0.00350000010803342,0.120212018489838,0.170682936906815,0.306500017642975,0.389499992132187,0.472547292709351,0.0124000012874603,0.00310030998662114,0.0015999999595806,0.0595999993383884,0.00620000017806888,0.243848770856857,0.159215912222862,0.367863208055496,0.362472504377365,0.360763907432556,0.0741000026464462,
0.00500000035390258,0.00250000017695129,0.162299990653992,0.232600003480911,0.308830857276917,0.273800015449524,0.0207000020891428,0.0412958711385727,0.0744074434041977,0.195099994540215,0.103310324251652,0.132200002670288,0.352999985218048,0.368699997663498,0.369100004434586,0.397839784622192,0.425842583179474,0.401159882545471,0.384400010108948,0.351700007915497,0.332100033760071,0.307400017976761,0.253174692392349,0.188718885183334,0.114722937345505,0.000499950023368001,0.000999900046736002,0.0239000003784895,0.00159984000492841,0.0450954921543598,0.109010890126228,0.159284085035324,0.197339460253716,0.238400012254715,0.140500009059906,0.249500006437302,0.256974309682846,0.303269684314728,0.200620055198669,0.178999990224838,0.197919800877571,0.270027011632919,0.113288670778275,0.296299993991852,0.351335138082504,0.371162921190262,0.16159999370575,0.318300008773804,0.228577151894569,0.448399990797043,0.455745577812195,0.455700010061264,0.454954504966736,0.0794999971985817,0.0401000007987022,0.108589150011539,0.0550054982304573,0.129712969064713,0.0664000064134598,0.135199993848801,0.0687137469649315,0.144800007343292,0.110511057078838,0.0705000013113022,0.099990002810955,0.138799995183945,0.179000005125999,0.219099998474121,0.274399995803833,0.121412143111229,0.00989999994635582,0.00490048993378878,0.170782938599586,0.234623461961746,0.301269859075546,0.366600006818771,0.407599985599518,0.364936500787735,0.383561611175537,0.386261343955994,0.168783128261566,0.0147000001743436,0.00730073032900691,0.179117918014526,0.236699998378754,0.293770641088486,0.379999995231628,0.446499973535538,0.360799998044968,0.369563043117523,0.346734702587128,0.300999999046326,0.175500005483627,0.100100003182888,0.00410041026771069,0.00250025023706257,0.00100000004749745,0.273999989032745,0.147900000214577,0.094700001180172,0.00229977001436055,0.00110011000651866,0.186981290578842,0.134886503219604,0.0750000029802322,0.0940999984741211,0.0218000002205372,0.000499999965541065,0.123400002717972,0.0233999993652105,0.0019000000320375,0.0118000004440546,
0.00479952013120055,0.000499950023368001,0.0059000002220273,0.00240000034682453,0.000200020003831014,0.386600017547607,0.317799985408783,0.298599988222122,0.252700001001358,0.314999997615814,0.300599992275238,0.2483000010252,0.129399999976158,0.260326027870178,0.204099997878075,0.186418637633324,0.0401000007987022,0.0176035221666098,0.000600000028498471,0.251000016927719,0.269600003957748,0.165283486247063,0.0414000004529953,0.224399998784065,0.188899993896484,0.146099999547005,0.0108989104628563,0.21647834777832,0.144199997186661,0.0639936029911041,0.0116999996826053,0.115711562335491,0.113600000739098,0.0214000009000301,0.0447999984025955,0.00670000026002526,0.129112914204597,0.0834999978542328,0.0434000007808208,0.00820000097155571,0.0883088335394859,0.0898000001907349,0.0109999999403954,0.043699998408556,0.0260973889380693,0.0784000009298325,0.0511999987065792,0.022299999371171,0.00410000002011657,0.060093991458416,0.063000001013279,0.00580000039190054,0.0324999988079071,0.0177999995648861,0.00699929986149073,0.0263000000268221,0.0186000000685453,0.0110000008717179,0.0020000000949949,0.029100002720952,0.0303000006824732,0.00290029007010162,0.0158984120935202,0.00879999995231628,0.00719999987632036,0.0273000001907349,0.013399999588728,0.0295000001788139,0.147499993443489,0.0217000003904104,0.0974097400903702,0.0247000027447939,0.217600002884865,0.206879317760468,0.143900007009506,0.0434999987483025,0.0288971103727818,0.0296970326453447,0.0301969796419144,0.0422957688570023,0.0484048388898373,0.046000000089407,0.0249000024050474,0.005899409763515,0.0180981904268265,0.139313936233521,0.114399999380112,0.0761076137423515,0.0366000011563301,0.156384363770485,0.000899909937288612,0.000699930009432137,0.000500000023748726,0.00120000005699694,0.00109999999403954,0.000600000028498471,0.000300000014249235,0.00960000045597553,0.0117000006139278,0.00920000020414591,0.0061006098985672,0.00300000002607703,0.00650000013411045,0.0183000005781651,0.0225000027567148,0.0199999995529652,0.0140000004321337,0.00810081046074629,0.0106000006198883,
0.0490049012005329,0.0696930289268494,0.059405941516161,0.0412000007927418,0.0229977015405893,0.00260000000707805,0.00510102044790983,0.00980098079890013,0.0274000000208616,0.0585058517754078,0.0742925703525543,0.0590000003576279,0.0397039763629436,0.0204000025987625,0.00270000007003546,0.00540054077282548,0.0103000001981854,0.0307000000029802,0.0661066174507141,0.0811081156134605,0.0525052510201931,0.0318999998271465,0.0136986300349236,0.00259974040091038,0.00700000021606684,0.0131013114005327,0.0342965684831142,0.0169016905128956,0.033100001513958,0.0445000045001507,0.0151000004261732,0.0384038425981998,0.0267973206937313,0.00679931975901127,0.00340034021064639,0.0135013507679105,0.00679931975901127,0.326367378234863,0.202420234680176,0.122000008821487,0.074800007045269,0.0441000014543533,0.0246975310146809,0.0123999994248152,0.407000005245209,0.44154417514801,0.44870001077652,0.450399994850159,0.331699997186661,0.188999995589256,0.102989710867405,0.0449954979121685,0.0407959185540676,0.0337000004947186,0.0236000008881092,0.0118000013753772,0.0838999971747398,0.056994304060936,0.0425000041723251,0.121799997985363,0.123099997639656,0.102989695966244,0.0674067363142967,0.0326000042259693,0.124099999666214,0.111200004816055,0.0306030604988337,0.0234999991953373,0.0122999995946884,0.11259999871254,0.105800002813339,0.0861999988555908,0.0491000041365623,0.0243999995291233,0.108989097177982,0.063900001347065,0.000600000028498471,0.000200020003831014,0.000300000043353066,0.000699930009432137
}
Transform: *16 {
a: -0.123373719334279,-0.99196749922193,-0.027920301987042,0,0.992355646027652,-0.123409809667645,-0.000433905912925563,0,-0.00301521838998339,-0.027760406427305,0.999610006806728,0,-143.172744832652,15.6358909641465,-2.8187465670338,1
}
TransformLink: *16 {
a: -0.123373739421368,0.992355763912201,-0.00301521876826882,0,-0.991967380692227,-0.123409802494671,-0.0277604032298401,0,-0.0279203050840802,-0.000433906111948186,0.999610125984453,0,-2.23216390609741,144.006698608398,2.8200089931488,1
}
}
Fine, so I know where my skin weights are located. This deformer is linked with a model belonging to my skeleton, like the below one:
Model: 43880864, "Model::clavicle_r", "LimbNode" {
Version: 232
Properties70: {
P: "RotationActive", "bool", "", "",1
P: "InheritType", "enum", "", "",1
P: "ScalingMax", "Vector3D", "Vector", "",0,0,0
P: "DefaultAttributeIndex", "int", "Integer", "",0
P: "Lcl Translation", "Lcl Translation", "", "A",-2.5624293094302,24.4011353780869,1.75273844028327
P: "Lcl Rotation", "Lcl Rotation", "", "A+",-1.59647649138759,0.126098006759319,97.8618363489333
P: "MHName", "KString", "", "U", "clavicle_r"
}
Culling: "CullingOff"
}
This model seems related to a PoseNode element:
PoseNode: {
Node: 43880864
Matrix: *16 {
a: -0.123373724520206,-0.991967499256134,-0.0279203020036221,-0,0.992355644702911,-0.123409815132618,-0.000433906068792567,-0,-0.00301521853543818,-0.0277604050934315,0.999610006809235,0,-143.172744750977,15.6358909606934,-2.81874656677246,1
}
}
Also there are several animation data linked to the above model, which are composed by an AnimationCurveNode, like this one...:
AnimationCurveNode: 41325152, "AnimCurveNode::R", "" {
Properties70: {
P: "d|X", "Number", "", "A",0
P: "d|Y", "Number", "", "A",0
P: "d|Z", "Number", "", "A",0
}
}
...And 3 AnimationCurve linked to the AnimationCurveNode, like the below example:
AnimationCurve: 43322352, "AnimCurve::", "" {
Default: 6.83911848068237
KeyVer: 4009
KeyTime: *30 {
a: 0,1539538600,3079077200,4618615800,6158154400,7697693000,9237231600,10776770200,12316308800,13855847400,15395386000,16934924600,18474463200,20014001800,21553540400,23093079000,24632617600,26172156200,27711694800,29251233400,30790772000,32330310600,33869849200,35409387800,36948926400,38488465000,40028003600,41567542200,43107080800,44646619400
}
KeyValueFloat: *30 {
a: 6.839118,4.812448,2.579699,2.663273,2.931432,0.4413721,-2.407341,-2.829627,-2.010718,-0.2999563,0.4959856,0.8600912,2.385289,7.922067,12.00326,14.10983,16.25888,17.10504,13.76837,11.45222,7.290313,4.249648,3.026491,4.031856,4.573201,5.226569,5.798617,6.815637,7.556128,6.839089
}
;KeyAttrFlags: Cubic|TangeantAuto|GenericTimeIndependent
KeyAttrFlags: *1 {
a: 8456
}
;KeyAttrDataFloat: RightAuto:0, NextLeftAuto:0
KeyAttrDataFloat: *4 {
a: 0,0,218434821,0
}
KeyAttrRefCount: *1 {
a: 30
}
}
As I said, I'm comfortable with several concepts in 3d animated models, so the weights and KeyTime/KeyData structures are pretty clear for me.
But I just cannot understand which data I should use to build my skeleton, as all the following ones may potentially do the job (or even perhaps none of them?):
Should I use the Transform matrix from the deformer?, or
Should I use the TransformLink instead?, or
Should I use the Lcl Translation/Lcl Rotation available in the model?, or
Should I use the matrix available in the pose node?, or
Should I use a mix of all these values?
And what are the difference e.g between Transform and TransformLink matrices?
In the same way, the available data for the animation let me also puzzled, as there is a float value which should be applied with the time... but to what? And what this value means, as it's obviously not a percentage (i.e between 0 and 1)?
If someone may explain me a little more in details what these values are meaning, and how I may use them, I would be greatfull.
So after a month of search and headache, I finally managed to understand how all these data were working together.
The "Lcl Translation" and "Lcl Rotation" contained in the Model structure represent a bone in the skeleton, once combined into a matrix. The models hierarchy is the skeleton itself.
The TransformLink matrix contained in the Deformer structure is the matrix to combine with the bone matrix, to transform each vertex to their final location. Note that this matrix should be inverted before be used.
Finally the AnimationCurve structures linked with a parent AnimationCurveNode contain the new values to apply to each bone matrix they are linked with. They contain the 3 x, y and z values which represent a new local translation or rotation over the time, where "AnimCurveNode::T" represents a translation while "AnimCurveNode::R" represents a rotation.
I couldn't find any usage for the Transform matrix contained in the Deformer structure, as well as for the PoseNode hierarchy.
If you're interested by the final result, my FBX reader project is available here: https://github.com/Jeanmilost/Demos/tree/master/Visual%20Studio/OpenGL/FBX

Accessing labels and confidence values with Tensorflow.js custom model?

(EDIT: partly solved, please see the end of the post) I have successfully uploaded a custom Tensorflow.js classifier that makes predictions of my webcam feed. It's trained on two classes, "good" and "bad". I get results as an array (correct me if I'm using the terms wrong as I'm a newbie!)
As the next step, I want to print the labels and confidence values on my browser. However, I find it a bit hard to grasp how I access my "good" and "bad" labels and confidence values from this result format that I get:
const result = await model.predict(t4d);
console.log(result);
(console)
t {kept: false, isDisposedInternal: false, shape: Array(2), dtype: "float32", size: 2, …}
dataId:
__proto__: Object
dtype: "float32"
id: 685817
isDisposedInternal: false
kept: false
rankType: "2"
scopeId: 883969
shape: (2) [1, 2]
size: 2
strides: [2]
isDisposed: (...)
rank: (...)
__proto__: Object
If I understand right, it gives me shape 2 = class 2 as a result (or is it just the shape and not my class yet?). Can anybody give advice on how to manually assign labels for these shapes 1 and 2 so that they can be printed? And how would I access confidence values - with argMax?
Ideally I would want it print something like this:
Good: 0.703
Bad: 0.298
Reference is this out-of-the-box mobilenet example from Tensorflow.js tutorials:
const webcam = await tf.data.webcam(webcamElement);
while (true) {
const img = await webcam.capture();
const result = await net.classify(img);
document.getElementById('console').innerText = `
prediction: ${result[0].className}\n
probability: ${result[0].probability}
`;
Where it prints the most probable class label and confidence value (like cat, 0.3444).
Warm thanks for any help!
EDIT: By following this tutorial, I was able to extract the label by making an array of my labels:
const labels = ["good", "bad"];

Is Alexa still sending the Viewport data in the Request?

I don't see the Viewport data neither from a Show or a Spot... (context.Viewport) What happened?
There used to be a section to describe the display capabilites, like this from a Spot:
"Viewport": {
"currentPixelHeight": 480,
"currentPixelWidth": 480,
"dpi": 160,
"experiences": [
{
"arcMinuteHeight": 144,
"arcMinuteWidth": 144,
"canResize": false,
"canRotate": false
}
],
"keyboard": [],
"pixelHeight": 480,
"pixelWidth": 480,
"shape": "ROUND",
"touch": [
"SINGLE"
]
}
Ok. Scratch that. It magically came back. Looks like Amazon is reading Stack Overflow.
I have been scouring documentation for days looking for a way to get the display characteristics. This post gave me a tiny clue, thank you very much. I finally found out how to get them with this. I know it seems a simple task to get the variable but I'm just beginning and the documentation isn't that friendly (or I was looking in the wrong places). So if one prints event['context']['Viewport'] then one can get the display characteristics when the test skill is run.
Here's my example for getting the screen width:
def get_display_width(event):
try:
width = event['context']['Viewport']['currentPixelWidth']
except:
width = 1
if width < 481:
return 1 #small hub round 480 x 480
elif width < 962:
return 2 #small hub 960 x 480
elif width < 1026:
return 3 #medium hub 1024 x 600
else:
return 4 #large hubs and tv's

sagemaker clienterror rows 1-5000 have more fields than expected size 3

I have created a K-means training job with a csv file that I have stored in S3. After a while I receive the following error:
Training failed with the following error: ClientError: Rows 1-5000 in file /opt/ml/input/data/train/features have more fields than than expected size 3.
What could be the issue with my file?
Here are the parameters I am passing to sagemaker.create_training_job
TrainingJobName=job_name,
HyperParameters={
'k': '2',
'feature_dim': '2'
},
AlgorithmSpecification={
'TrainingImage': image,
'TrainingInputMode': 'File'
},
RoleArn='arn:aws:iam::<my_acc_number>:role/MyRole',
OutputDataConfig={
"S3OutputPath": output_location
},
ResourceConfig={
'InstanceType': 'ml.m4.xlarge',
'InstanceCount': 1,
'VolumeSizeInGB': 20,
},
InputDataConfig=[
{
'ChannelName': 'train',
'ContentType': 'text/csv',
"CompressionType": "None",
"RecordWrapperType": "None",
'DataSource': {
'S3DataSource': {
'S3DataType': 'S3Prefix',
'S3Uri': data_location,
'S3DataDistributionType': 'FullyReplicated'
}
}
}
],
StoppingCondition={
'MaxRuntimeInSeconds': 600
}
I've seen this issue appear when doing unsupervised learning, such as the above example using clustering. If you have a csv input, you can also address this issue by setting label_size=0 in the ContentType parameter of the Sagemaker API call, within the InputDataConfig branch.
Here's an example of what the relevant section of the call might look like:
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "some/path/in/s3",
"S3DataDistributionType": "ShardedByS3Key"
}
},
"CompressionType": "None",
"RecordWrapperType": "None",
"ContentType": "text/csv;label_size=0"
}
]
Make sure your .csv doesn't have column headers, and that the label is the first column.
Also make sure your values for the hyper-parameters are accurate ie feature_dim means number of features in your set. If you give it the wrong value, it'll break.
Heres a list of sagemaker knn hyper-parameters and their meanings: https://docs.aws.amazon.com/sagemaker/latest/dg/kNN_hyperparameters.html

ext js 4 column chart bug? series remain visible when I hide them

Feeling I had not enough control over the chart if I had used a grouped column chart, I made my own version by just adding different series to the chart. After all the store, the number of series, their colors and such all need to be set dynamically and not hard coded. So basically this is what I have:
chart = Ext.create("Ext.chart.Chart", {
store: dataStore,
axes: dynamicAxes,
series: series
});
I leave out the not interesting stuff such as width, height of the chart etc.
now I have a method whichs returns a series object. This is added to the series array mentioned in the code above. The function has a "item" object parameter and also an idx param which is the index of the item object from the array it comes from, and a max param which is the size of the item array
the function returns something like this:
var w = (typeof (max) !== "undefined" && max !== null) ? this._getWidthByMax(max) : 30;
return {
type: "column",
axis = "left",
xField = "timestamp",
yField = item.id, // store field name equals the id of the item object
style = { stroke: colorCode, "stroke-width": (item.isDefault) ? 2 : 1, fill: colorCode },
width = w,
renderer = function (sprite, rec, attr, bix) {
var nx = idx * w;
return Ext.apply(attr, { translation: { x: nx} });
}
}
now this works fine for the number of columns I want to have. That can be one, two, three... up to seven currently.
However, if I want to hide a series, the following call doesn't work:
chart.series.getAt(idx).hideAll();
while it does work if I render my chart as a line chart.
is this a bug in Ext-js 4 or is it because of how I rendered the series for my column chart?
since nobody has replied to my question and I have found a solution in the meantime, I might as well answer my own question...
the problem occurred in Ext Js 4.0.7.
With version 4.1 RC 2 the hideAll behaved correctly.
So the solution, for anyone who would have the same problem, is to upgrade to 4.1 RC 2 or newer.

Resources