Important
You are viewing documentation for an older version of Confluent Platform. For the latest, click here.
Line-Delimited Source Connector¶
This connector is used to read a file line by line and write the data to Kafka.
To use this connector, use a connector configuration that specifies the name of this connector class in the connector.class configuration property:
connector.class=com.github.jcustenborder.kafka.connect.spooldir.SpoolDirLineDelimitedSourceConnector
Important
The recommended converter to use is the StringConverter. Example: value.converter=org.apache.kafka.connect.storage.StringConverter
The other connector-specific configuration properties are described below.
FIX Encoded Lines Example¶
This example reads files in a directory line by line and parse them using kafka-connect-transform-fix to a FIX representation of the data.
Configuration¶
{
"connector.class" : "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirLineDelimitedSourceConnector",
"topic" : "fix",
"input.path" : "/tmp",
"input.file.pattern" : "^.+\\.fix$",
"error.path" : "/tmp",
"finished.path" : "/tmp",
"transforms" : "fromFix",
"transforms.fromFix.type" : "com.github.jcustenborder.kafka.connect.transform.fix.FromFIX$Value"
}
Data similar to the following will be emitted by this connector.
Output¶
{
"sourcePartition" : { },
"sourceOffset" : { },
"topic" : "fix",
"kafkaPartition" : 0,
"valueSchema" : {
"name" : "fix42.NewOrderSingle",
"type" : "STRUCT",
"isOptional" : false,
"fieldSchemas" : {
"Account" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "1"
},
"isOptional" : true
},
"CashOrderQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "152"
},
"isOptional" : true
},
"CheckSum" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "10"
},
"isOptional" : true
},
"ClOrdID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "11"
},
"isOptional" : true
},
"ClearingAccount" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "440"
},
"isOptional" : true
},
"ClearingFirm" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "439"
},
"isOptional" : true
},
"ClientID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "109"
},
"isOptional" : true
},
"CommType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "13"
},
"isOptional" : true
},
"Commission" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "12"
},
"isOptional" : true
},
"ComplianceID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "376"
},
"isOptional" : true
},
"ContractMultiplier" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "231"
},
"isOptional" : true
},
"CouponRate" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "223"
},
"isOptional" : true
},
"CoveredOrUncovered" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "203"
},
"isOptional" : true
},
"Currency" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "15"
},
"isOptional" : true
},
"CustomerOrFirm" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "204"
},
"isOptional" : true
},
"DiscretionInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "388"
},
"isOptional" : true
},
"DiscretionOffset" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "389"
},
"isOptional" : true
},
"EffectiveTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "168"
},
"isOptional" : true
},
"EncodedIssuer" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "349"
},
"isOptional" : true
},
"EncodedIssuerLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "348"
},
"isOptional" : true
},
"EncodedSecurityDesc" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "351"
},
"isOptional" : true
},
"EncodedSecurityDescLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "350"
},
"isOptional" : true
},
"EncodedText" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "355"
},
"isOptional" : true
},
"EncodedTextLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "354"
},
"isOptional" : true
},
"ExDestination" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "100"
},
"isOptional" : true
},
"ExecBroker" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "76"
},
"isOptional" : true
},
"ExecInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "18"
},
"isOptional" : true
},
"ExpireDate" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "432"
},
"isOptional" : true
},
"ExpireTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "126"
},
"isOptional" : true
},
"ForexReq" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "121"
},
"isOptional" : true
},
"FutSettDate" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "64"
},
"isOptional" : true
},
"FutSettDate2" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "193"
},
"isOptional" : true
},
"GTBookingInst" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "427"
},
"isOptional" : true
},
"HandlInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "21"
},
"isOptional" : true
},
"IDSource" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "22"
},
"isOptional" : true
},
"IOIID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "23"
},
"isOptional" : true
},
"Issuer" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "106"
},
"isOptional" : true
},
"LocateReqd" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "114"
},
"isOptional" : true
},
"MaturityDay" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "205"
},
"isOptional" : true
},
"MaturityMonthYear" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "200"
},
"isOptional" : true
},
"MaxFloor" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "111"
},
"isOptional" : true
},
"MaxShow" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "210"
},
"isOptional" : true
},
"MinQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "110"
},
"isOptional" : true
},
"NoAllocs" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "78"
},
"isOptional" : true
},
"NoTradingSessions" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "386"
},
"isOptional" : true
},
"OpenClose" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "77"
},
"isOptional" : true
},
"OptAttribute" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "206"
},
"isOptional" : true
},
"OrdType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "40"
},
"isOptional" : true
},
"OrderQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "38"
},
"isOptional" : true
},
"OrderQty2" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "192"
},
"isOptional" : true
},
"PegDifference" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "211"
},
"isOptional" : true
},
"PrevClosePx" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "140"
},
"isOptional" : true
},
"Price" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "44"
},
"isOptional" : true
},
"ProcessCode" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "81"
},
"isOptional" : true
},
"PutOrCall" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "201"
},
"isOptional" : true
},
"QuoteID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "117"
},
"isOptional" : true
},
"Rule80A" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "47"
},
"isOptional" : true
},
"SecurityDesc" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "107"
},
"isOptional" : true
},
"SecurityExchange" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "207"
},
"isOptional" : true
},
"SecurityID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "48"
},
"isOptional" : true
},
"SecurityType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "167"
},
"isOptional" : true
},
"SettlCurrency" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "120"
},
"isOptional" : true
},
"SettlmntTyp" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "63"
},
"isOptional" : true
},
"Side" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "54"
},
"isOptional" : true
},
"Signature" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "89"
},
"isOptional" : true
},
"SignatureLength" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "93"
},
"isOptional" : true
},
"SolicitedFlag" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "377"
},
"isOptional" : true
},
"StopPx" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "99"
},
"isOptional" : true
},
"StrikePrice" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "202"
},
"isOptional" : true
},
"Symbol" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "55"
},
"isOptional" : true
},
"SymbolSfx" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "65"
},
"isOptional" : true
},
"Text" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "58"
},
"isOptional" : true
},
"TimeInForce" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "59"
},
"isOptional" : true
},
"TransactTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "60"
},
"isOptional" : true
}
}
},
"value" : {
"schema" : {
"name" : "fix42.NewOrderSingle",
"type" : "STRUCT",
"isOptional" : false,
"fieldSchemas" : {
"Account" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "1"
},
"isOptional" : true
},
"CashOrderQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "152"
},
"isOptional" : true
},
"CheckSum" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "10"
},
"isOptional" : true
},
"ClOrdID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "11"
},
"isOptional" : true
},
"ClearingAccount" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "440"
},
"isOptional" : true
},
"ClearingFirm" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "439"
},
"isOptional" : true
},
"ClientID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "109"
},
"isOptional" : true
},
"CommType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "13"
},
"isOptional" : true
},
"Commission" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "12"
},
"isOptional" : true
},
"ComplianceID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "376"
},
"isOptional" : true
},
"ContractMultiplier" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "231"
},
"isOptional" : true
},
"CouponRate" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "223"
},
"isOptional" : true
},
"CoveredOrUncovered" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "203"
},
"isOptional" : true
},
"Currency" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "15"
},
"isOptional" : true
},
"CustomerOrFirm" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "204"
},
"isOptional" : true
},
"DiscretionInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "388"
},
"isOptional" : true
},
"DiscretionOffset" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "389"
},
"isOptional" : true
},
"EffectiveTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "168"
},
"isOptional" : true
},
"EncodedIssuer" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "349"
},
"isOptional" : true
},
"EncodedIssuerLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "348"
},
"isOptional" : true
},
"EncodedSecurityDesc" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "351"
},
"isOptional" : true
},
"EncodedSecurityDescLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "350"
},
"isOptional" : true
},
"EncodedText" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "355"
},
"isOptional" : true
},
"EncodedTextLen" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "354"
},
"isOptional" : true
},
"ExDestination" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "100"
},
"isOptional" : true
},
"ExecBroker" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "76"
},
"isOptional" : true
},
"ExecInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "18"
},
"isOptional" : true
},
"ExpireDate" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "432"
},
"isOptional" : true
},
"ExpireTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "126"
},
"isOptional" : true
},
"ForexReq" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "121"
},
"isOptional" : true
},
"FutSettDate" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "64"
},
"isOptional" : true
},
"FutSettDate2" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "193"
},
"isOptional" : true
},
"GTBookingInst" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "427"
},
"isOptional" : true
},
"HandlInst" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "21"
},
"isOptional" : true
},
"IDSource" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "22"
},
"isOptional" : true
},
"IOIID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "23"
},
"isOptional" : true
},
"Issuer" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "106"
},
"isOptional" : true
},
"LocateReqd" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "114"
},
"isOptional" : true
},
"MaturityDay" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "205"
},
"isOptional" : true
},
"MaturityMonthYear" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "200"
},
"isOptional" : true
},
"MaxFloor" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "111"
},
"isOptional" : true
},
"MaxShow" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "210"
},
"isOptional" : true
},
"MinQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "110"
},
"isOptional" : true
},
"NoAllocs" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "78"
},
"isOptional" : true
},
"NoTradingSessions" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "386"
},
"isOptional" : true
},
"OpenClose" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "77"
},
"isOptional" : true
},
"OptAttribute" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "206"
},
"isOptional" : true
},
"OrdType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "40"
},
"isOptional" : true
},
"OrderQty" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "38"
},
"isOptional" : true
},
"OrderQty2" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "192"
},
"isOptional" : true
},
"PegDifference" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "211"
},
"isOptional" : true
},
"PrevClosePx" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "140"
},
"isOptional" : true
},
"Price" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "44"
},
"isOptional" : true
},
"ProcessCode" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "81"
},
"isOptional" : true
},
"PutOrCall" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "201"
},
"isOptional" : true
},
"QuoteID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "117"
},
"isOptional" : true
},
"Rule80A" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "47"
},
"isOptional" : true
},
"SecurityDesc" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "107"
},
"isOptional" : true
},
"SecurityExchange" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "207"
},
"isOptional" : true
},
"SecurityID" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "48"
},
"isOptional" : true
},
"SecurityType" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "167"
},
"isOptional" : true
},
"SettlCurrency" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "120"
},
"isOptional" : true
},
"SettlmntTyp" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "63"
},
"isOptional" : true
},
"Side" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "54"
},
"isOptional" : true
},
"Signature" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "89"
},
"isOptional" : true
},
"SignatureLength" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "93"
},
"isOptional" : true
},
"SolicitedFlag" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "377"
},
"isOptional" : true
},
"StopPx" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "99"
},
"isOptional" : true
},
"StrikePrice" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "202"
},
"isOptional" : true
},
"Symbol" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "55"
},
"isOptional" : true
},
"SymbolSfx" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "65"
},
"isOptional" : true
},
"Text" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "58"
},
"isOptional" : true
},
"TimeInForce" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "59"
},
"isOptional" : true
},
"TransactTime" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "60"
},
"isOptional" : true
}
}
},
"fieldValues" : [ {
"name" : "Account",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "1"
},
"isOptional" : true
}
}, {
"name" : "CashOrderQty",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "152"
},
"isOptional" : true
}
}, {
"name" : "CheckSum",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "10"
},
"isOptional" : true
}
}, {
"name" : "ClOrdID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "11"
},
"isOptional" : true
},
"storage" : "NF 0542/03232009"
}, {
"name" : "ClearingAccount",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "440"
},
"isOptional" : true
}
}, {
"name" : "ClearingFirm",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "439"
},
"isOptional" : true
}
}, {
"name" : "ClientID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "109"
},
"isOptional" : true
}
}, {
"name" : "CommType",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "13"
},
"isOptional" : true
}
}, {
"name" : "Commission",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "12"
},
"isOptional" : true
}
}, {
"name" : "ComplianceID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "376"
},
"isOptional" : true
}
}, {
"name" : "ContractMultiplier",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "231"
},
"isOptional" : true
}
}, {
"name" : "CouponRate",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "223"
},
"isOptional" : true
}
}, {
"name" : "CoveredOrUncovered",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "203"
},
"isOptional" : true
}
}, {
"name" : "Currency",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "15"
},
"isOptional" : true
}
}, {
"name" : "CustomerOrFirm",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "204"
},
"isOptional" : true
}
}, {
"name" : "DiscretionInst",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "388"
},
"isOptional" : true
}
}, {
"name" : "DiscretionOffset",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "389"
},
"isOptional" : true
}
}, {
"name" : "EffectiveTime",
"schema" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "168"
},
"isOptional" : true
}
}, {
"name" : "EncodedIssuer",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "349"
},
"isOptional" : true
}
}, {
"name" : "EncodedIssuerLen",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "348"
},
"isOptional" : true
}
}, {
"name" : "EncodedSecurityDesc",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "351"
},
"isOptional" : true
}
}, {
"name" : "EncodedSecurityDescLen",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "350"
},
"isOptional" : true
}
}, {
"name" : "EncodedText",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "355"
},
"isOptional" : true
}
}, {
"name" : "EncodedTextLen",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "354"
},
"isOptional" : true
}
}, {
"name" : "ExDestination",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "100"
},
"isOptional" : true
}
}, {
"name" : "ExecBroker",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "76"
},
"isOptional" : true
}
}, {
"name" : "ExecInst",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "18"
},
"isOptional" : true
}
}, {
"name" : "ExpireDate",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "432"
},
"isOptional" : true
}
}, {
"name" : "ExpireTime",
"schema" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "126"
},
"isOptional" : true
}
}, {
"name" : "ForexReq",
"schema" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "121"
},
"isOptional" : true
}
}, {
"name" : "FutSettDate",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "64"
},
"isOptional" : true
}
}, {
"name" : "FutSettDate2",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "193"
},
"isOptional" : true
}
}, {
"name" : "GTBookingInst",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "427"
},
"isOptional" : true
}
}, {
"name" : "HandlInst",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "21"
},
"isOptional" : true
},
"storage" : "1"
}, {
"name" : "IDSource",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "22"
},
"isOptional" : true
}
}, {
"name" : "IOIID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "23"
},
"isOptional" : true
}
}, {
"name" : "Issuer",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "106"
},
"isOptional" : true
}
}, {
"name" : "LocateReqd",
"schema" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "114"
},
"isOptional" : true
}
}, {
"name" : "MaturityDay",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "205"
},
"isOptional" : true
}
}, {
"name" : "MaturityMonthYear",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "200"
},
"isOptional" : true
}
}, {
"name" : "MaxFloor",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "111"
},
"isOptional" : true
}
}, {
"name" : "MaxShow",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "210"
},
"isOptional" : true
}
}, {
"name" : "MinQty",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "110"
},
"isOptional" : true
}
}, {
"name" : "NoAllocs",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "78"
},
"isOptional" : true
}
}, {
"name" : "NoTradingSessions",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "386"
},
"isOptional" : true
}
}, {
"name" : "OpenClose",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "77"
},
"isOptional" : true
}
}, {
"name" : "OptAttribute",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "206"
},
"isOptional" : true
}
}, {
"name" : "OrdType",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "40"
},
"isOptional" : true
},
"storage" : "1"
}, {
"name" : "OrderQty",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "38"
},
"isOptional" : true
},
"storage" : 100.0
}, {
"name" : "OrderQty2",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "192"
},
"isOptional" : true
}
}, {
"name" : "PegDifference",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "211"
},
"isOptional" : true
}
}, {
"name" : "PrevClosePx",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "140"
},
"isOptional" : true
}
}, {
"name" : "Price",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "44"
},
"isOptional" : true
}
}, {
"name" : "ProcessCode",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "81"
},
"isOptional" : true
}
}, {
"name" : "PutOrCall",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "201"
},
"isOptional" : true
}
}, {
"name" : "QuoteID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "117"
},
"isOptional" : true
}
}, {
"name" : "Rule80A",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "47"
},
"isOptional" : true
},
"storage" : "A"
}, {
"name" : "SecurityDesc",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "107"
},
"isOptional" : true
}
}, {
"name" : "SecurityExchange",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "207"
},
"isOptional" : true
},
"storage" : "N"
}, {
"name" : "SecurityID",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "48"
},
"isOptional" : true
}
}, {
"name" : "SecurityType",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "167"
},
"isOptional" : true
}
}, {
"name" : "SettlCurrency",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "120"
},
"isOptional" : true
}
}, {
"name" : "SettlmntTyp",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "63"
},
"isOptional" : true
}
}, {
"name" : "Side",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "54"
},
"isOptional" : true
},
"storage" : "1"
}, {
"name" : "Signature",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "89"
},
"isOptional" : true
}
}, {
"name" : "SignatureLength",
"schema" : {
"type" : "INT32",
"parameters" : {
"fix.field" : "93"
},
"isOptional" : true
}
}, {
"name" : "SolicitedFlag",
"schema" : {
"type" : "BOOLEAN",
"parameters" : {
"fix.field" : "377"
},
"isOptional" : true
}
}, {
"name" : "StopPx",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "99"
},
"isOptional" : true
}
}, {
"name" : "StrikePrice",
"schema" : {
"type" : "FLOAT64",
"parameters" : {
"fix.field" : "202"
},
"isOptional" : true
}
}, {
"name" : "Symbol",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "55"
},
"isOptional" : true
},
"storage" : "CVS"
}, {
"name" : "SymbolSfx",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "65"
},
"isOptional" : true
}
}, {
"name" : "Text",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "58"
},
"isOptional" : true
}
}, {
"name" : "TimeInForce",
"schema" : {
"type" : "STRING",
"parameters" : {
"fix.field" : "59"
},
"isOptional" : true
},
"storage" : "0"
}, {
"name" : "TransactTime",
"schema" : {
"name" : "org.apache.kafka.connect.data.Timestamp",
"type" : "INT64",
"version" : 1,
"parameters" : {
"fix.field" : "60"
},
"isOptional" : true
},
"storage" : 1237822829000
} ]
},
"headers" : [ ]
}
Configuration Properties¶
General¶
topicThe Kafka topic to write the data to.
- Importance: HIGH
- Type: STRING
batch.sizeThe number of records that should be returned with each batch.
- Importance: LOW
- Type: INT
- Default Value: 1000
empty.poll.wait.msThe amount of time to wait if a poll returns an empty list of records.
- Importance: LOW
- Type: LONG
- Default Value: 250
- Validator: [1,…,9223372036854775807]
File System¶
error.pathThe directory to place files that have errors. This directory must exist and be writable by the user running Kafka Connect.
- Importance: HIGH
- Type: STRING
- Validator: Absolute path to a directory that exists and is writable.
input.file.patternRegular expression to check input file names against. This expression must match the entire filename. The equivalent of
Matcher.matches().- Importance: HIGH
- Type: STRING
input.pathThe directory where Kafka Connect reads files that are processed. This directory must exist and be writable by the user running Connect.
- Importance: HIGH
- Type: STRING
- Validator: Absolute path to a directory that exists and is writable.
finished.pathThe directory where Connect puts files that are successfully processed. This directory must exist and be writable by the user running Connect.
- Importance: HIGH
- Type: STRING
halt.on.errorSets whether the task halts when it encounters an error or continues to the next file.
- Importance: HIGH
- Type: BOOLEAN
- Default Value: true
cleanup.policySets how the connector should clean up files that are successfully processed.
NONEleaves the files in place. Files left in place may be reprocessed if the connector is restarted.DELETEremoves the file from the filesystem.MOVE(the default) moves the file to thefinished.pathdirectory.- Importance: MEDIUM
- Type: STRING
- Default Value: MOVE
- Validator: Matches:
NONE,DELETE,MOVE
file.minimum.age.msThe amount of time in milliseconds after the file was last written to before the file can be processed.
- Importance: LOW
- Type: LONG
- Default Value: 0
- Validator: [0,…]
processing.file.extensionBefore a file is processed, it is renamed to indicate that it is currently being processed. This setting is appended to the end of the file.
- Importance: LOW
- Type: STRING
- Default Value: .PROCESSING
- Validator: Matches regex( ^.*..+$ )
Timestamps¶
timestamp.modeDetermines how the connector sets the timestamp for the ConnectRecord. If set to
Field, the timestamp is read from a field in the value. This field cannot be optional and must be a Timestamp. Specify the field intimestamp.field. If set toFILE_TIME, the last time the file was modified is used. If set toPROCESS_TIME(the default), the time the record is read is used.- Importance: MEDIUM
- Type: STRING
- Default Value: PROCESS_TIME
- Validator: Matches:
FIELD,FILE_TIME,PROCESS_TIME