I'm generating a arff file with groovy from a xslx,
but when i try to open this file in weka i got this error:
File "..." not recognised as an 'Arff data files' file.
Reason:
nominal value not declared in header, read Token[Ativo], line 16
i can't understand why i'm getting this error
can someone helpme to fix this error, and explain why it's happening?
Generated file
#relation kd-itempedido
#attribute tipopedido {Assistencia,Recompra,Venda,Troca}
#attribute aprovado {0.0,1.0}
#attribute fasepedido {Aprovado,Cancelado,EmAprovacao,Liberado,Novo}
#attribute statusinternopedido {NegociarPagamento,PedidosDeTeste,AguardandoOcorrencia,Nada,AguardandoBoletoDeposito,PedidoDuplicado,SuspeitaDeFraude}
#attribute canal {Marketplace,Desktop}
#attribute origem {LojasAmericanas,Optimise,MercadoLivre,Cityads,Zanox,Zoom,Rakuten,Lomadee,Facebook,Viptarget,Submarino,Criteo,Muccashop,Chaordic,Walmart,Googlead,Nada,Extra,Lojaskd,Shopback,Afilio,Shoptime,Nextperformance,CarrinhoAbandonado,Bing}
#attribute mercado {S,N}
#attribute cluster {EntregaImediata,Fiprec,Icconv,Esgotado}
#attribute statusitem {Ativo}
#attribute statusproduto {Inativo,Ativo,AtivoSemEstoque,ForaDeLinha}
#attribute polo {Polo1,Polo3,Polo2}
#data
Venda,0.0,Novo,Nada,Desktop,Googlead,S,Fiprec,Ativo,Ativo,Polo2
Venda,0.0,Novo,Nada,Desktop,Googlead,S,Fiprec,Ativo,Ativo,Polo2
Venda,0.0,Novo,Nada,Desktop,Googlead,S,Ativo,Inativo,Polo2
Venda,0.0,Novo,Nada,Desktop,Muccashop,N,Ativo,Ativo,Polo3
Groovy (VM -Dfile.encoding=ascii utf-8 utf8)
#Grapes([
#Grab('org.apache.poi:poi:3.10.1'),
#Grab('org.apache.poi:poi-ooxml:3.10.1')])
import org.apache.poi.xssf.usermodel.XSSFWorkbook
import java.text.Normalizer
import static org.apache.poi.ss.usermodel.Cell.*
import java.nio.file.Paths
def path = "/home/eric/Documents/development/ufpr/Solid Eric/ItemPedido1000.xlsx"
def relation = "kd-itempedido"
def columns = ["tipopedido", "aprovado", "fasepedido", "statusinternopedido", "canal", "origem", "mercado", "cluster", "statusitem","statusproduto", "polo"]
def arff = "ItemPedido.arff"
new XslxToArffParser(path, relation, columns, arff);
class Data{
def rows = new ArrayList<List>();
#Override
String toString() {
def s = ""
for (r in rows){
for(d in r){
s+=d
if(r.indexOf(d) < (r.size()-1))
s+=","
}
s+="\n"
}
return s
}
}
class Atributo {
def descricao;
def possibilidades = new HashSet<Object>();
def index;
#Override
String toString() {
def builder = new StringBuilder()
builder.append("#attribute ").append(descricao)
builder.append(" {")
for(def i = 0; i<possibilidades.size(); i++){
builder.append(possibilidades[i])
if((i+1) != possibilidades.size())
builder.append(",")
}
builder.append("}").append("\n")
return builder.toString();
}
}
class XslxToArffParser {
def attributes =[:];
def data = new Data();
def sheet = null;
XslxToArffParser(path, relation, columns, arffPath){
load(path)
getAttributes(columns)
collectData()
saveArff(relation, arffPath)
}
def String parse(String s){
s = Normalizer.normalize(s, Normalizer.Form.NFD)
s = s.replaceAll("[\\p{InCombiningDiacriticalMarks}]", "")
s = s.split(/[^\w]/).collect { it.toLowerCase().capitalize() }.join("")
s = s.replaceAll(" ", "")
s = s.replaceAll("[^A-Za-z0-9]", "")
s = s.isEmpty() ? "Nada" : s
return s
}
def load(path) {
Paths.get(path).withInputStream { input ->
def workbook = new XSSFWorkbook(input)
sheet = workbook.getSheetAt(0)
}
}
def getAttributes(columns){
for (cell in sheet.getRow(0).cellIterator()) {
def index = cell.columnIndex
def description = parse(cell.stringCellValue).toLowerCase()
if(columns.contains(description)){
attributes << [(index):new Atributo(descricao: description, index: index)]
}
}
}
def collectData(){
def headerFlag = true
for (row in sheet.rowIterator()) {
if (headerFlag) {
headerFlag = false
continue
}
def r = []
for (cell in row.cellIterator()) {
def index = cell.columnIndex;
def value = cell.cellType == CELL_TYPE_STRING ? parse(cell.stringCellValue) : cell.numericCellValue
def attr = attributes[index]
if(attr != null){
attr.possibilidades.add(value)
r << value
}
}
data.rows.add(r)
}
}
def saveArff(relation, path){
Paths.get(path).withWriter { writer ->
writer.write "#relation " + relation
writer.write "\n"
for(a in attributes.values())
writer.write a.toString()
writer.write "#data"
writer.write "\n"
writer.write data.toString()
}
}
}
Solved. "row.cellIterator()" does not iterate over null/blank cells
It has been a while since I used Weka, but looking at the file you showed and the error message, I suspect the problem is in the last two rows of the data file. They don't have a value for the attribute "cluster".
After the S or N (for attribute "mercado"), they have "Ativo". That "Ativo" value is not defined as one of the possible values of the nominal attribute cluster. The file did read "Ativo" though (which is why the error message says ''read Token[Ativo]'', but it expected to read a value for the cluster attribute, it did not yet expect a value for the statusitem attribute.
I want to adapt my Java Spark app (which actually uses RDDs for some calculations) to use Datasets instead of RDDs. I'm new to Datasets and not sure how to map which transaction to a corresponding Dataset operation.
At the moment I map them like this:
JavaSparkContext.textFile(...) -> SQLContext.read().textFile(...)
JavaRDD.filter(Function) -> Dataset.filter(FilterFunction)
JavaRDD.map(Function) -> Dataset.map(MapFunction)
JavaRDD.mapToPair(PairFunction) -> Dataset.groupByKey(MapFunction) ???
JavaPairRDD.aggregateByKey(U, Function2, Function2) -> KeyValueGroupedDataset.???
And the corresponing questions are:
Equals JavaRDD.mapToPair the Dataset.groupByKey method?
Does JavaPairRDD map to KeyValueGroupedDataset?
Which method equals the JavaPairRDD.aggregateByKey method?
However, I want to port the following RDD code into a Dataset one:
JavaRDD<Article> goodRdd = ...
JavaPairRDD<String, Article> ArticlePairRdd = goodRdd.mapToPair(new PairFunction<Article, String, Article>() { // Build PairRDD<<Date|Store|Transaction><Article>>
public Tuple2<String, Article> call(Article article) throws Exception {
String key = article.getKeyDate() + "|" + article.getKeyStore() + "|" + article.getKeyTransaction() + "|" + article.getCounter();
return new Tuple2<String, Article>(key, article);
}
});
JavaPairRDD<String, String> transactionRdd = ArticlePairRdd.aggregateByKey("", // Aggregate distributed data -> PairRDD<String, String>
new Function2<String, Article, String>() {
public String call(String oldString, Article newArticle) throws Exception {
String articleString = newArticle.getOwg() + "_" + newArticle.getTextOwg(); // <<Date|Store|Transaction><owg_textOwg###owg_textOwg>>
return oldString + "###" + articleString;
}
},
new Function2<String, String, String>() {
public String call(String a, String b) throws Exception {
String c = a.concat(b);
...
return c;
}
}
);
My code looks this yet:
Dataset<Article> goodDS = ...
KeyValueGroupedDataset<String, Article> ArticlePairDS = goodDS.groupByKey(new MapFunction<Article, String>() {
public String call(Article article) throws Exception {
String key = article.getKeyDate() + "|" + article.getKeyStore() + "|" + article.getKeyTransaction() + "|" + article.getCounter();
return key;
}
}, Encoders.STRING());
// here I need something similar to aggregateByKey! Not reduceByKey as I need to return another data type (String) than I have before (Article)
We have multiple Typewriter .tst templates in our project, and would like to share some common logic/methods between them. Is there a way to do this?
You can use T4 templates to generate the Typewriter templates. Put the code inside a reusable T4 template (*.ttinclude) and create tt-files to pass parameters to the rendering method of this base template.
(I use a Visual Studio extension for File nesting.)
Each tt-file looks something like this;
<## template debug="true" hostSpecific="true" #>
<## output extension=".tst" #>
<## include file="..\ModelsTemplate.ttinclude" #>
<# this.ModelsTemplate_Render("UserSettings"); #>
...and my ttinclude file looks like this (it is a bit project specific, but I included it all so that anyone who wants to try it out can easily get something working);
<## IntelliSenseLanguage processor="tangibleT4Editor" language="C#" #>
<#+
void ModelsTemplate_Render(string #subnamespace) {
ModelsTemplate_Render("MyApp.ViewData", #subnamespace);
}
void ModelsTemplate_Render(string #mainnamespace, string #subnamespace) {
string renderedMainNamespace = #mainnamespace;
#>
// <auto-generated>
// This code was generated by a tool.
// Template: <#= Host.TemplateFile #>
// Changes to this file may cause incorrect behavior and will be lost if
// the code is regenerated.
// </auto-generated>
${
// Enable extension methods by adding using Typewriter.Extensions.*
using Typewriter.Extensions.Types;
using System.Text.RegularExpressions;
Template(Settings settings)
{
settings.IncludeProject("MyApp.ViewData");
}
static string DebugInfo = "";
string PrintDebugInfo(File f){
if (string.IsNullOrEmpty(DebugInfo)) {
return "";
}
return "/*" + Environment.NewLine + "Template debug info: " + DebugInfo + Environment.NewLine + "*/";
}
string BaseClassFullPath(Class baseClass)
{
//DebugInfo = DebugInfo + Environment.NewLine + "baseClass.FullName:" + baseClass.FullName;
var result = baseClass.Name;
// Until we find a better way to handle implementations of generic base classes...
result = result.Replace("<bool?>", "<boolean>");
result = result.Replace("<int?>", "<number>");
result = result.Replace("<long?>", "<number>");
result = result.Replace("<decimal?>", "<number>");
result = result.Replace("<double?>", "<number>");
result = result.Replace("<System.Double>", "<number>");
result = result.Replace("<System.Double?>", "<number>");
result = result.Replace("<System.DateTime?>", "<Date>");
return result;
}
string NullableFilter(string typeName)
{
return typeName.Replace("?", "");
}
string TypeFilteredPath(Type type)
{
//DebugInfo = DebugInfo + Environment.NewLine + "type:" + type.FullName + " - genericType:" + type.Unwrap().FullName;
return NullableFilter(type.Name);
}
string TypeFiltered(Type type)
{
if (type.IsEnumerable)
{
var genericType = type.Unwrap();
if (!genericType.FullName.StartsWith("System"))
{
return TypeFilteredPath(genericType)+"[]";
}
}
return TypeFilteredPath(type);
}
string PropertyTypeFiltered(Property prop)
{
return TypeFiltered(prop.Type);
}
string ImplementedInterfaces(Class c){
if (!c.Interfaces.Any()){
return string.Empty;
}
return "implements " + string.Join(", ", c.Interfaces.Select(x => x.FullName));
}
string ExtendedInterfaces(Interface i){
if (!i.Interfaces.Any()){
return string.Empty;
}
return "extends " + string.Join(", ", i.Interfaces.Select(x => x.FullName));
}
string DescriptionAttributeValue(Attribute a){
if (!a.FullName.Contains("DescriptionAttribute")){
return string.Empty;
}
return a.Value;
}
string GetPropertyDefinitionWithScope(Property p){
var definition = GetPropertyDefinition(p);
if (definition != "")
return "public " + definition;
else
return definition;
}
string GetPropertyDefinition(Property p){
var ignoreAttribute = p.Attributes.SingleOrDefault(x => x.FullName.Contains("TypeScriptIgnoreMemberAttribute"));
if (ignoreAttribute != null)
return "";
var typeAttribute = p.Attributes.SingleOrDefault(x => x.FullName.Contains("TypeScriptTypeAttribute"));
if (typeAttribute != null) {
return p.name + ": " + typeAttribute.Value + ";";
}
return p.name + ": " + TypeFiltered(p.Type) + ";";
}
string WriteImports(Class theClass){
//return "import { ViewDataEntity } from '../common/ViewDataEntity';";
var list = new List<string>();
var typesToImport = theClass.Properties.Select(x => x.Type.Unwrap()).Where(x => x.Namespace.Contains("MyApp.ViewData.")).ToList();
if (theClass.BaseClass?.Namespace.Contains("MyApp.ViewData.") == true)
typesToImport.Add(theClass.BaseClass);
foreach (var impType in typesToImport)
{
var modules = impType.Namespace.Replace("MyApp.ViewData.", "").Split('.').ToList();
string modPart = string.Join("/", modules.Select(x => CamelCase(x)));
list.Add($"import {{ {impType.Name} }} from '../{modPart}/{impType.Name}';");
}
return string.Join(Environment.NewLine, list.Distinct());
}
string CamelCase(string value){
return value.First().ToString().ToLower() + value.Substring(1);
}
}//namespace <#= renderedMainNamespace #>.<#= #subnamespace #> {
$Classes(c => c.Namespace.StartsWith("<#= #mainnamespace #>.<#= #subnamespace #>"))[
$WriteImports
export class $Name$TypeParameters $BaseClass[extends $BaseClassFullPath ] $ImplementedInterfaces {$Properties[
$GetPropertyDefinitionWithScope]
}]
$Interfaces(<#= #mainnamespace #>.<#= #subnamespace #>.*)[
export interface $Name $ExtendedInterfaces {
$Properties[
$GetPropertyDefinition]
}]
$Enums(<#= #mainnamespace #>.<#= #subnamespace #>.*)[
export class $Name {
$Values[// $Value - "$Attributes[$Value]"
static $Name = "$Name";
]
}]
$PrintDebugInfo
//}<#+
}
#>
Unfortunately there's no way to share code in tst templates. Support for this will probably come in a future version though.
I am working around with Salesforce and force.com API and metadata API, version 36.
I can create a custom field in a Lead object but by default I can see it's hidden and this means I cannot create a new Lead with these custom fields because it returns a bad request (400 status code).
Is there any way by Code to set the custom field Visible?
public boolean createCustomExtTextField(String name, LoginResult metadataLoginResult, int length) {
boolean success = false;
CustomField cs = new CustomField();
cs.setFullName("Lead."+name+"__c");
cs.setLabel("Custom"+name+"Field");
cs.setType(FieldType.LongTextArea);
cs.setLength(length);
cs.setVisibleLines(50); // max 50
try {
MetadataConnection metadataConnection = createMetadataConnection(metadataLoginResult);
SaveResult[] results = metadataConnection.createMetadata(new Metadata[] { cs });
for (SaveResult r : results) {
if (r.isSuccess()) {
success = true;
} else {
System.out.println("Errors were encountered while creating " + r.getFullName());
for (com.sforce.soap.metadata.Error e : r.getErrors()) {
System.out.println("Error message: " + e.getMessage());
System.out.println("Status code: " + e.getStatusCode());
}
}
}
} catch (ConnectionException e) {
e.printStackTrace();
}
return success;
}
I am googling a lot and don't find something that actually helped. So, any hints are welcomed. Thank you.
Finally found a solution to this. I final one for me was to make all custom fields REQUIRED.
CustomField cs = new CustomField();
cs.setFullName("Lead.YourCompanyName" + name + "__c");
cs.setLabel("YourCompanyName" + name);
cs.setRequired(true);
...
com.sforce.soap.enterprise.LoginResult metadataLoginResult = operations.loginToMetadata(username, password, "https://login.salesforce.com/services/Soap/c/36.0");
...
private boolean createFieldInMetadata(LoginResult metadataLoginResult, CustomField cs) {
boolean success = false;
try {
MetadataConnection metadataConnection = createMetadataConnection(metadataLoginResult);
SaveResult[] results = metadataConnection.createMetadata(new Metadata[] { cs });
for (SaveResult r : results) {
if (r.isSuccess()) {
success = true;
} else {
System.out.println("Errors were encountered while creating " + r.getFullName());
for (com.sforce.soap.metadata.Error e : r.getErrors()) {
System.out.println("Error message: " + e.getMessage());
System.out.println("Status code: " + e.getStatusCode());
}
}
}
} catch (Exception e) {
}
return success;
}
And so it will appear in the page layout. Very important to know, a required field cannot have just an empty value set, it must be something. So if not all custom fields are required in your logic and you wanna avoid the entire process of unzipping page layout and zipping it back (however it may be done) just add "N/A" or any char at choice to the required by code but not your project custom fields.
I managed to make the custom Field Level Security visible for "Admin" profile but not Field Accessability to visible. The latter is untested.
Please note: I know for the question SolrNet - The given key was not present in the dictionary and I have initialized solr object just like Mauricio suggests.
I am using solr 4.6.0 and solrnet build #173, .net framework 4.0 and VS2012 for development. For some unknown reason I am receiving error 'The given key was not present in the dictionary'. I have a document with that id in solr, I've checked via browser. It's a document like any other document. Why is error popping up? My code (I've made a comment on the place where the error happens):
//establishes connection with solr
private void ConnectToSolr()
{
try
{
if (_solr != null) return;
Startup.Init<Register>(SolrAddress);
_solr = ServiceLocator.Current.GetInstance<ISolrOperations<Register>>();
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
}
//Returns snippets from solr as BindingSource
public BindingSource GetSnippets(string searchTerm, DateTime? startDate = null, DateTime? endDate = null)
{
ConnectToSolr();
string dateQuery = startDate == null
? ""
: endDate == null
? "savedate:\"" + convertDateToSolrFormat(startDate) + "\"" //only start date
: "savedate:[" + convertDateToSolrFormat(startDate) + " TO " +
convertDateToSolrFormat(endDate) + "]";//range between start and end date
string textQuery = string.IsNullOrEmpty(searchTerm) ? "text:*" : "text:*" + searchTerm + "*";
List<Register> list = new List<Register>();
SolrQueryResults<Register> results;
string currentId = "";
try
{
results = _solr.Query(textQuery,
new QueryOptions
{
Highlight = new HighlightingParameters
{
Fields = new[] { "*" },
},
ExtraParams = new Dictionary<string, string>
{
{"fq", dateQuery},
{"sort", "savedate desc"}
}
});
for (int i = 0; i < results.Highlights.Count; i++)
{
currentId = results[i].Id;
var h = results.Highlights[currentId];
if (h.Snippets.Count > 0)
{
list.Add(new Register//here the error "the given key was not present in the dictionary pops up
{
Id = currentId,
ContentLiteral = h.Snippets["content"].ToArray()[0].Trim(new[]{' ', '\n'}),
SaveDateLiteral = results[i].SaveDate.ToShortDateString()
});
}
}
BindingList<Register> bindingList = new BindingList<Register>(list);
BindingSource bindingSource = new BindingSource();
bindingSource.DataSource = bindingList;
return bindingSource;
}
catch(Exception e)
{
MessageBox.Show(string.Format("{0}\nId:{1}", e.Message, currentId), "Solr error");
return null;
}
}
I've found out what's causing the problem: saving empty documents into solr. If I make an empty query (with text:*) through solrnet (usually I do this if I want to see all saved documents) and empty document is one of saved docs, then 'The given key is not present in dictionary pops up'. If all of the documents have text in them, this error doesn't pop up.
If you document contains fields with types other than string and you index null value to a double or integer field you will get the same error.
solr query return the null field as:
<null name="fieldname"/>
should be
<double name="fieldname">0.0</double>
or
<double name="fieldname"/>