Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
pulsar
提交
0724482f
pulsar
项目概览
apache
/
pulsar
通知
129
Star
40
Fork
3
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Wiki
1
Wiki
分析
仓库
DevOps
项目成员
Pages
pulsar
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Pages
分析
分析
仓库分析
DevOps
Wiki
1
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
提交
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
未验证
提交
0724482f
编写于
8月 24, 2020
作者:
B
Boyang Jerry Peng
提交者:
GitHub
8月 24, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add ability for BatchPushSource to notify errors asynchronously (#7865)
Co-authored-by:
N
Jerry Peng
<
jerryp@splunk.com
>
上级
b00f5e79
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
164 addition
and
0 deletion
+164
-0
pulsar-io/batch-data-generator/src/main/java/org/apache/pulsar/io/batchdatagenerator/BatchDataGeneratorPushSource.java
...r/io/batchdatagenerator/BatchDataGeneratorPushSource.java
+81
-0
pulsar-io/core/src/main/java/org/apache/pulsar/io/core/BatchPushSource.java
.../main/java/org/apache/pulsar/io/core/BatchPushSource.java
+26
-0
pulsar-io/core/src/test/java/org/apache/pulsar/io/core/BatchPushSourceTest.java
...t/java/org/apache/pulsar/io/core/BatchPushSourceTest.java
+57
-0
未找到文件。
pulsar-io/batch-data-generator/src/main/java/org/apache/pulsar/io/batchdatagenerator/BatchDataGeneratorPushSource.java
0 → 100644
浏览文件 @
0724482f
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.pulsar.io.batchdatagenerator
;
import
io.codearte.jfairy.Fairy
;
import
lombok.extern.slf4j.Slf4j
;
import
org.apache.pulsar.functions.api.Record
;
import
org.apache.pulsar.io.core.BatchPushSource
;
import
org.apache.pulsar.io.core.SourceContext
;
import
java.util.Map
;
import
java.util.concurrent.ExecutorService
;
import
java.util.concurrent.Executors
;
import
java.util.function.Consumer
;
@Slf4j
public
class
BatchDataGeneratorPushSource
extends
BatchPushSource
<
Person
>
implements
Runnable
{
private
Fairy
fairy
;
private
SourceContext
sourceContext
;
private
int
maxRecordsPerCycle
=
10
;
private
ExecutorService
executor
=
Executors
.
newSingleThreadExecutor
();
@Override
public
void
close
()
{
executor
.
shutdownNow
();
}
@Override
public
void
open
(
Map
config
,
SourceContext
context
)
throws
Exception
{
this
.
fairy
=
Fairy
.
create
();
this
.
sourceContext
=
context
;
}
@Override
public
void
discover
(
Consumer
taskEater
)
throws
Exception
{
log
.
info
(
"Generating one task for each instance"
);
for
(
int
i
=
0
;
i
<
sourceContext
.
getNumInstances
();
++
i
)
{
taskEater
.
accept
(
String
.
format
(
"something-%d"
,
System
.
currentTimeMillis
()).
getBytes
());
}
}
@Override
public
void
prepare
(
byte
[]
instanceSplit
)
throws
Exception
{
log
.
info
(
"Instance "
+
sourceContext
.
getInstanceId
()
+
" got a new discovered task {}"
,
new
String
(
instanceSplit
));
executor
.
submit
(
this
);
}
@Override
public
void
run
()
{
try
{
for
(
int
i
=
0
;
i
<
maxRecordsPerCycle
;
i
++)
{
Thread
.
sleep
(
50
);
Record
<
Person
>
record
=
()
->
new
Person
(
fairy
.
person
());
consume
(
record
);
}
// this task is completed
consume
(
null
);
}
catch
(
Exception
e
)
{
notifyError
(
e
);
}
}
}
pulsar-io/core/src/main/java/org/apache/pulsar/io/core/BatchPushSource.java
浏览文件 @
0724482f
...
...
@@ -37,6 +37,21 @@ public abstract class BatchPushSource<T> implements BatchSource<T> {
}
}
private
static
class
ErrorNotifierRecord
implements
Record
{
private
Exception
e
;
public
ErrorNotifierRecord
(
Exception
e
)
{
this
.
e
=
e
;
}
@Override
public
Object
getValue
()
{
return
null
;
}
public
Exception
getException
()
{
return
e
;
}
}
private
LinkedBlockingQueue
<
Record
<
T
>>
queue
;
private
static
final
int
DEFAULT_QUEUE_LENGTH
=
1000
;
private
final
NullRecord
nullRecord
=
new
NullRecord
();
...
...
@@ -48,6 +63,9 @@ public abstract class BatchPushSource<T> implements BatchSource<T> {
@Override
public
Record
<
T
>
readNext
()
throws
Exception
{
Record
<
T
>
record
=
queue
.
take
();
if
(
record
instanceof
ErrorNotifierRecord
)
{
throw
((
ErrorNotifierRecord
)
record
).
getException
();
}
if
(
record
instanceof
NullRecord
)
{
return
null
;
}
else
{
...
...
@@ -80,4 +98,12 @@ public abstract class BatchPushSource<T> implements BatchSource<T> {
public
int
getQueueLength
()
{
return
DEFAULT_QUEUE_LENGTH
;
}
/**
* Allows the source to notify errors asynchronously
* @param ex
*/
public
void
notifyError
(
Exception
ex
)
{
consume
(
new
ErrorNotifierRecord
(
ex
));
}
}
\ No newline at end of file
pulsar-io/core/src/test/java/org/apache/pulsar/io/core/BatchPushSourceTest.java
0 → 100644
浏览文件 @
0724482f
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.pulsar.io.core
;
import
org.apache.pulsar.io.core.BatchPushSource
;
import
org.apache.pulsar.io.core.SourceContext
;
import
org.testng.annotations.Test
;
import
java.util.Map
;
import
java.util.function.Consumer
;
public
class
BatchPushSourceTest
{
BatchPushSource
testBatchSource
=
new
BatchPushSource
()
{
@Override
public
void
open
(
Map
config
,
SourceContext
context
)
throws
Exception
{
}
@Override
public
void
discover
(
Consumer
taskEater
)
throws
Exception
{
}
@Override
public
void
prepare
(
byte
[]
task
)
throws
Exception
{
}
@Override
public
void
close
()
throws
Exception
{
}
};
@Test
(
expectedExceptions
=
RuntimeException
.
class
,
expectedExceptionsMessageRegExp
=
"test exception"
)
public
void
testNotifyErrors
()
throws
Exception
{
testBatchSource
.
notifyError
(
new
RuntimeException
(
"test exception"
));
testBatchSource
.
readNext
();
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录